Page 1 of 2 12 LastLast
Results 1 to 30 of 35

Thread: Best practical archiver

  1. #1
    Member
    Join Date
    Jan 2007
    Location
    Moscow
    Posts
    239
    Thanks
    0
    Thanked 3 Times in 1 Post
    I'd want you to discuss the subject.

    Speaking about all parts of archiver - compression strength, compression and decompression speed, UI, reliability, new technologies (like SSE, multi-threading etc.) usage, resonable memory usage and so on.

    I think that there is no ideal archiver, but my best ones are WinRAR and 7-Zip. Their strengths are: best GUI, powerfull CLI versions, latest CPU optimizations, multithread support, most spreaded archives support, large tweaking capabilities, good default settings. Besides 7-Zip has support for large memory amounts.

    WinRAR is weak in small dictionary size, so it doesn use most of the memory, i think it's due to back compatibility.

    My best nowadays is 7-Zip. Using fast compression, 16 mb dictionary and 16 word size it beats WinRAR both in speed and compression most of the time. But it has relatively (in contrast to WinRAR, uharc, sbc) weak multimedia support and doesn't support compression algorithm change in one archive during compression. Other bad property is very bad (in terms of speed) compression of already compressed data. Most algorithms, except LZP and DEFLATE, just get stuck and drop compression speed significantly.
    Furthermore i'd like to see "thor ex" (as speedy) and "uharc mz" (as fast) algorithms included in 7-zip. I know that LZP has relatively small decompression speed, but it doesn't matter for me.

    One more interesting archiver i'd like to mention is FreeArc (http://freearc.narod.ru/). It uses PPM, LZMA, BWT and LZP algoritms but, unfortunately, doesn't change them dynamically in one archive.

    So i want to ask: can anyone complete one practical archiver, which will use rising power of recent machines more effectively? Yes, it's cool to invent new algoritms and methods, but most people will never use your brilliant programs because of low speed and compatibility uncertainty. If someone will make new archiver or complete 7-Zip it will be awesome! And please remember - you can hardly find a computer with less then 256 mb memory, most have 512 and more.

    Thanks.

  2. #2
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by nimdamsk
    can anyone complete one practical archiver, which will use rising power of recent machines more effectively?
    This is never possible, IT are developing rapidly... you can either have backward compatible archiver or archiver with outstanding compression power, not both. In other words, there will always be people with state of the art hardware, who will want the best possible programs, and there will be people with quite old hardware, who wont be able to use such programs.

    Some combinations are possible via programs such as KGB or PeaZip, which use programs powerful enough and wrap them in easy-to-use GUI

  3. #3
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    Quote Originally Posted by nimdamsk
    Yes, its cool to invent new algoritms and methods, but most people will never use your brilliant programs because of low speed and compatibility uncertainty.
    However, the download statistics of PIM arhciver and my web site statistics tell the different thing. Well, I just interested in data compression and new algorithms. I dont care about how much people use my programs. Low speed? I dont think so. Compatibility issues? And again PIM archiver is really keeps full backward compatibility at least. Compress any file with first version of PIM and you will able to unpack it with the newest version. Same thing about the new QUAD engine – next 1.08BETA is faster and more reliable and at the same time fully compatible with 1.07BETA...

  4. #4
    Member
    Join Date
    Jan 2007
    Location
    Moscow
    Posts
    239
    Thanks
    0
    Thanked 3 Times in 1 Post
    I appreciate hard work of encode and other people on the scene. They move it forward surely. But please answer, why do so many people write so much compressors nowadays?
    Are they strongest at compression? No. Undoubtedly leaders are PAQ-like algorithms with CM and neuro-tricks. As for WinRK, IMHO it's main power are good filters, not compression algoritm, may be i am wrong.
    I don't want to criticize anyone, but here is quick'n'dirty test i've just made. I compressed full installed NERO folder with different archivers.

    Folders 151
    Files 1923
    Files size 410,775,058

    7z │ 144,899 K - 154 sec - 16mb dict, 16 word size
    rar │ 149,009 K - 173 sec - maximum
    uha│ 152,323 K - 116 sec - 32mb dict, mz
    SBC│ 156,479 K - 184 sec - b9
    pim│ 177,427 K - 359 sec - default
    zip│ 192,518 K - 83 sec - maximum in WinRAR

    So, no comments. Yes, i am not testing guru, but nevertheless, it's a typical situation. I also understand that 7-zip is been developed several years (earlier as BIX, 777 and UFA), and that you can further enchance your programs.

    I think that Ilia improved his algoritms greatly and discovered some new ones. Can you turn from tuning heavy CM-like algoritms to fast BWT and LZxx? I'd like to see opensource THOR competitor. You can also tune LZX algorithm, it has great decompression speed.

  5. #5
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    The testing results are heavily depends on data, since each algorithm has a strong and a weak sides. I hope you and others understand that.

    Quote Originally Posted by nimdamsk
    Can you turn from tuning heavy CM-like algoritms to fast BWT and LZxx?
    Well, I have my own line of algorithms, own area of interests. I dont like BWT. The latest QUAD, which Im working now, is LZ-based.

    Quote Originally Posted by nimdamsk
    Id like to see opensource THOR competitor.
    IMHO, THOR is too fast with too weak compression. For fast compression/decompression coders nowadays Ill recommend something LZH-based, like Deflate (ZIP) and RAR in Fast/Fastest mode.

    Quote Originally Posted by nimdamsk
    You can also tune LZX algorithm, it has great decompression speed.
    LZX is good, but it already invented and well implemented – no reason to make another clone. Again, I have my own ideas and even if my programs will be somewhere in the middle, its okay.


  6. #6
    Member
    Join Date
    Jan 2007
    Location
    Moscow
    Posts
    239
    Thanks
    0
    Thanked 3 Times in 1 Post
    Quote Originally Posted by encode
    IMHO, THOR is too fast with too weak compression. For fast compression/decompression coders nowadays Ill recommend something LZH-based, like Deflate (ZIP) and RAR in Fast/Fastest mode.
    May be. Lets check I archived Nero folder into zero-compressed 7z archive.

    ex │ 179,588 K - 26 sec - thor ex
    zip│ 191,109 K - 97 sec - WinRAR max compression
    zip│ 200,760 K - 34 sec - WinRAR max speed
    │ 401,171 K - original

    And, for reference

    pim│ 157,816 K - 436 sec

    Again, its a typical situation, ive made such test many times.

    And NERO archive is a good example. Its biggest 200 files (361,093,668 bytes from full 410,775,058 bytes) are:
    258,969,380 bytes in 152 files - PE 32 executables
    58,245,952 (37) - almost uncompressible 7z, img, jpg, mpg, nct, png, VDB, zip
    40,740,448 bytes in 8 files - bmp
    2,789,728 bytes in 2 files - misc. binary
    There are also 1723 small (~30 kb each) files

    Can you tune LZSS? Isnt it interesting to make fastest practical archiver? I know you can do it

  7. #7
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    Dont forget, THOR has a symmetric compression (except with exx mode) - the decompression speed is the same or even slower compared to the compression.

    Quote Originally Posted by nimdamsk
    Isnt it interesting to make fastest practical archiver?
    Archiver is the crazy thing. PIM is a pretty simple archiver, but source code is already too complex - its HUGE. So, actually, I have no idea for future improvements. Probably, the latest update with new EXE-filter and QUAD deompressor is the final essential update of this archiver.

    Someday, I will have no spare time for data compression. Again, probably QUAD is the last thing. So, these days, Im hardly working on QUAD, making code faster and more reliable.

    LZSS? I started a data compression trip from simple LZSS compressor. Somewhere in the middle I turned to LZP, ROLZ and PPM...

    A very sentimental and philosophic topic...

  8. #8
    Member
    Join Date
    Jan 2007
    Location
    Moscow
    Posts
    239
    Thanks
    0
    Thanked 3 Times in 1 Post
    Quote Originally Posted by encode
    Dont forget, THOR has a symmetric compression (except with exx mode) - the decompression speed is the same or even slower compared to the compression.
    With thors speed it doesnt matter
    Quote Originally Posted by encode
    LZSS? I started a data compression trip from simple LZSS compressor. Somewhere in the middle I turned to LZP, ROLZ and PPM...
    I know that compression is your hobby only. Sadly, as i see, you dont want to make practical archiver, being interested in inventing new compression chemes. So, there are two ways - to sit and wait that one day someone (I. Pavlov?) will make 7-zip perfect or to make it myself. The problem is i cant But it can ve solved anyway.

    Thank you again for your work.

  9. #9
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    about FreeArc:
    1) my program homepage now is http://www.haskell.org/bz
    2) it divides all the files compressed into the text and binary groups and uses lzma for binary and ppmd for text files. also, i've added a bit of preprocessing techniques, you can see the whole compression algorithm by using --display option

    in *my* own tests, program works better than 7-zip, and is close to the uharc -mx. unfortunately, winrk craches in most of my tests (probably because i typically compress thousands of files), but i agree that winrk has best compression, on cost of speed

    mmx/sse can't be used in compression algorithms. multithreading, otoh, is the hottest topic now. i think it will be especially useful for improving ppmd speed because ppmd actually splits all the data in 6-7 mb blocks (with optimum settings), which are compressed independently. moreover, ppmd don't improve its compression when more than, say, 200 mb are available

    for best comression one should use -m5..-m8 options (depending on memory available, -m5 - 256megs, -m8 - 2gigs)

    alternatively, you can use -m5p..-m8p if external ppmonstr.exe is available on your system (it will be required for decompression too!)

    about MM compression - it's a myth. MM compression can be used only for *decompressed* audio/picture data - are you really have this? it can't compress mp3/avi/jpg files

  10. #10
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    about THOR: it uses lzh algorithm which is of course assymmetric. it uses large dictionary with a short search chain, so it works better than 32kb-dictionary zip or rar -m1. it's my beloving algorithm too. if his author will not release his algorithm under GPL license, once i will try to develop something like this

    meanwhile, try freearc with -m2 option. in my tests, it beats uharc -mz

    -m3 beats rar. -m4 beats 7-zip

  11. #11
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    FreeArc is very promising, but has no GUI...

    Quote Originally Posted by Bulat Ziganshin
    mmx/sse cant be used in compression algorithms.
    MMX works fine with PAQ8. However, with QUAD MMX/SSE not really works. With some experimental versions of QUAD I parallelize many loops using SSE - this gives no speed improvements - speed becomes even slower.

    Quote Originally Posted by Bulat Ziganshin
    about MM compression - its a myth. MM compression can be used only for *decompressed* audio/picture data - are you really have this? it cant compress mp3/avi/jpg files
    Its not a myth. BMP and WAV files are both contains multimedia data. Some users keep audio files in WAV (PCM) files and pictures in BMP (lossless). Your program just hasnt MM compression. It is worth to have at least simple delta filter for audio files - isnt it? Furthermore, many latest games have WAV file in their distribution. Why not to use MM compression? mp3/avi/jpg - are also multimedia files, but theyre already compressed - why compress twice??

  12. #12
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    Quote Originally Posted by Bulat Ziganshin
    about THOR: it uses lzh algorithm which is of course assymmetric.
    Nope. Only with exx. With others THOR uses LZP+HUFFMAN. As author said.

  13. #13
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Hello Bulat and welcome to the forums! Are you the one who developed Rep preprocessor?
    Quote Originally Posted by Bulat Ziganshin
    [MM compression] cant compress mp3/avi/jpg files
    mp3s could be precompressed to wav with MM applied to these then, couldnt they?

  14. #14
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    >Nope. Only with exx. With others THOR uses LZP+HUFFMAN. As author said.

    10x! i was in doubt whether one should use lzh or lzph for fast compression. now i will look for second alternative. are you know lzp preprocessor from grzip? it may be further improved to reach thor-like compression and speed

    >Are you the one who developed Rep preprocessor?

    yes. although it's very like rzip and lzp preprocessor, i only added a few ideas. but where you can hear about it?

    >mp3s could be precompressed to wav with MM applied to these then, couldn't they?

    if you mean that mp3s can be *decompressed* to wav - yes, but *lossless* MM compression will be worse than original lossy mp3 compression

    are you ever tried to compress avi/mp3/jpg with a MM archivers - rar, ace, uharc, winrk? that results are you got?

  15. #15
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    >FreeArc is very promising, but has no GUI...

    it seems that hardcore compression users want 1) compression, 2) CLI features, and only 3) GUI. and for me developing GUI would be rather hard - i don't had delphi-like RAD tools

    >MMX works fine with PAQ8.

    but not in practical algorithms. users often believe that some over-advertised technology like MM compression or SSE8.5 can improve their programs i don't know places in my current algorithms where mmx/sse can be used at all

    >It's not a myth. BMP and WAV files

    we can ask this partiicular user whether he really has uncompressed media files 12 years ago arjz was second program that included MM compression in its arsenal. but any feature need a time to develop and i don't want to spend my resources to work hard on real MM algorithms nor want to include simple delta just to say that my program includes MM compression. i just ask users why they need this. alongside of game and raw photo compression, i don't heared practical reasons

    ps: i'm really amazed how fast i got answers here! it seems that noone sleeps and all watch this forum 24h a day thank you, Ilia, for establishing such lovely place for communication!

  16. #16
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Bulat Ziganshin
    yes. although its very like rzip and lzp preprocessor, i only added a few ideas. but where you can hear about it?
    here and here

    Quote Originally Posted by Bulat Ziganshin
    mp3s can be *decompressed* to wav - yes, but *lossless* MM compression will be worse than original lossy mp3 compression
    Oh, I know this is completely impossible. I expressed my thoughts wrong... if MP3 is Huffman coded, then it can be partially decoded and coded with some stronger algorithm again (Soundslimmer, also StuffIt uses something similar for packing JPGs) - this could be an example of useful MM

  17. #17
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    >Soundslimmer

    of course, but it is not what is already implemented in other archivers. in particular, Soundslimmer was several months of work of very experienced programmers. i don't have enough skills to repeat their result. the same applies to jpeg compression. i guess that you know all these bells and whistles

    but i'm open to ideas of improving compression without hard work. good open-source libs that i can add to the program or easy-to-implement ideas. now i has ecm and ability to run external compressors in my list

    >here and here

    thank you! i will ask Ed for the file that shows up REP problems

  18. #18
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    By the way, why Haskell?

  19. #19
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    you may aks better - why archiver? when i started to learn Haskell, i has not found better program to implement using it. with all compression algorithms preimlemented in C, i can develop only rather high-level code and it's the area where Haskell rocks

    otherwise, i leaved this business 5 years ago and don't had plans to return to it

  20. #20
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    I have just a little knowledge about Haskell. So, is it possible to compile FreeArc under Linux? I bet CL programs are much popular under *nix platforms.

  21. #21
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    i never tried. but i just got suse and will try it. probably i will make arc unix-compatible due this month (and it will become best unix archiver because now p7-zip is best and my programs uses same lzma/ppmd with addiional tricks

    btw, now it uses rep, lzp, dict and bcj preprocessors (7zip uses bcj2, otoh), supports automatic selection of compression algorithm on file extension (but one need to write all the textfile extensions in arc.groups files, otherwise files will be compresed as binary), provides advanced file sorting algorithm. so, it's a great shell for compression algorithms. i tried not to not develop any compression algorithms anymore, but sometime has written two small preprocessors, dict and rep, to solve rather easy problems that was not solved by anyone else

    so i hope to further develop program in the direction of inclusion other's algorithms, adding small preprocessors and enhancing overall program usability

  22. #22
    Member Fallon's Avatar
    Join Date
    May 2008
    Location
    Europe - The Netherlands
    Posts
    158
    Thanks
    14
    Thanked 10 Times in 5 Posts
    Quote Originally Posted by encode
    FreeArc is very promising, but has no GUI...
    Quote Originally Posted by Bulat Ziganshin
    it seems that hardcore compression users want 1) compression, 2) CLI features, and only 3) GUI. and for me developing GUI would be rather hard - i dont had delphi-like RAD tools
    Okay, its hard. And if you program for hardcore compression users, thats better than for nobody!
    But to reach a wider audience, good compression needs a GUI. There is no way around it.

    Shortlist of general GUI requirements:
    1 - a GUI should loose little time, as most compression jobs are just a necessary waste of time.
    2 - a GUI has to be very(!) intuitive, meaning: as-few-clicks-as-possible to get to any feature, the shortest route to any command, in terms of time, should survive.
    3 - features should be useful.
    4 - for many features, the option to put/group them in profiles, maintains top intuitivity.
    5 - integration with popular GUIs is ok, starting with OS rightclick menu.
    Quote Originally Posted by Bulat Ziganshin
    i just got suse and will try it. probably i will make arc unix-compatible due this month (and it will become best unix archiver
    Best of luck with further development.

  23. #23
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    >But to reach a wider audience, good compression needs a GUI. There is no way around it.

    i know. just want to mention that if i started with great gui, then implemented features, and then goes into implementing compression, it will be even worser

  24. #24
    Member
    Join Date
    Mar 2007
    Location
    Ghent
    Posts
    56
    Thanks
    0
    Thanked 0 Times in 0 Posts
    For me this is a nobrainer: 7-Zip, and it has yet to include real multimedia filters!

    I hope Igor hasn't dropped the idea, It was once on his todo list.

  25. #25
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,984
    Thanks
    377
    Thanked 352 Times in 140 Posts
    А вообще по поводу "Бест":
    На вкус и цвет товарища нет!

    About the "best" term:
    [Russian proverb, partially untranslatable Russian folk]
    There is no friend for the same taste and color!

  26. #26
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    >For me this is a nobrainer: 7-Zip, and it has yet to include real multimedia filters!

    are you mean here bmp/wav or jpg/mp3 compression?

  27. #27
    Member
    Join Date
    Mar 2007
    Location
    Ghent
    Posts
    56
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Bulat Ziganshin
    are you mean here bmp/wav or jpg/mp3 compression?
    Any uncompressed image or audio data.

  28. #28
    Member
    Join Date
    Mar 2007
    Location
    Ghent
    Posts
    56
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Try 7-Zip, Quad, RAR, UHARC, WinRK ROLZ3 and CCM on this file:
    http://www.mediafire.com/?2ddtddq3dtl

  29. #29
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    it's better to say what it is the type of data and where such files are used

  30. #30
    Member
    Join Date
    Mar 2007
    Location
    Ghent
    Posts
    56
    Thanks
    0
    Thanked 0 Times in 0 Posts
    1.000.000 bytes of wav data without header:
    http://www.mediafire.com/?1y3zj3j5dmm

    680,106 wav.uha
    680,915 wav.ccm
    688,862 wav.paq8l
    695,184 wav.sbc
    696,073 wav.rar
    803,265 wav.mcomp
    893,424 wav.hook
    929,062 wav.7z
    938,003 wav.sqx
    954,961 wav.quad
    969,378 wav.zip
    1,000,000 wav

    Here you can clearly see UHARC's superior generic wave data compression.
    For example Squeez performs really bad here because it checks on extension or header to perform it's specialized wav routines.

Page 1 of 2 12 LastLast

Similar Threads

  1. Most efficient/practical compression method for short strings?
    By never frog in forum Data Compression
    Replies: 6
    Last Post: 1st September 2009, 05:05
  2. Maximum Practical Compression
    By Bulat Ziganshin in forum Forum Archive
    Replies: 5
    Last Post: 31st March 2008, 16:20
  3. Replies: 11
    Last Post: 20th February 2008, 11:39

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •