Best practical archiver

[nimdamsk]

I'd want you to discuss the subject.

Speaking about all parts of archiver - compression strength, compression and decompression speed, UI, reliability, new technologies (like SSE, multi-threading etc.) usage, resonable memory usage and so on.

I think that there is no ideal archiver, but my best ones are WinRAR and 7-Zip. Their strengths are: best GUI, powerfull CLI versions, latest CPU optimizations, multithread support, most spreaded archives support, large tweaking capabilities, good default settings. Besides 7-Zip has support for large memory amounts.

WinRAR is weak in small dictionary size, so it doesn use most of the memory, i think it's due to back compatibility.

My best nowadays is 7-Zip. Using fast compression, 16 mb dictionary and 16 word size it beats WinRAR both in speed and compression most of the time. But it has relatively (in contrast to WinRAR, uharc, sbc) weak multimedia support and doesn't support compression algorithm change in one archive during compression. Other bad property is very bad (in terms of speed) compression of already compressed data. Most algorithms, except LZP and DEFLATE, just get stuck and drop compression speed significantly.
Furthermore i'd like to see "thor ex" (as speedy) and "uharc mz" (as fast) algorithms included in 7-zip. I know that LZP has relatively small decompression speed, but it doesn't matter for me.

One more interesting archiver i'd like to mention is FreeArc (http://freearc.narod.ru/). It uses PPM, LZMA, BWT and LZP algoritms but, unfortunately, doesn't change them dynamically in one archive.

So i want to ask: can anyone complete one practical archiver, which will use rising power of recent machines more effectively? Yes, it's cool to invent new algoritms and methods, but most people will never use your brilliant programs because of low speed and compatibility uncertainty. If someone will make new archiver or complete 7-Zip it will be awesome! And please remember - you can hardly find a computer with less then 256 mb memory, most have 512 and more.

Thanks.

[Black_Fox1]

can anyone complete one practical archiver, which will use rising power of recent machines more effectively?

This is never possible, IT are developing rapidly... you can either have backward compatible archiver or archiver with outstanding compression power, not both. In other words, there will always be people with state of the art hardware, who will want the best possible programs, and there will be people with quite old hardware, who wont be able to use such programs.

Some combinations are possible via programs such as KGB or PeaZip, which use programs powerful enough and wrap them in easy-to-use GUI

[encode]

Yes, its cool to invent new algoritms and methods, but most people will never use your brilliant programs because of low speed and compatibility uncertainty.

However, the download statistics of PIM arhciver and my web site statistics tell the different thing. Well, I just interested in data compression and new algorithms. I dont care about how much people use my programs. Low speed? I dont think so. Compatibility issues? And again PIM archiver is really keeps full backward compatibility at least. Compress any file with first version of PIM and you will able to unpack it with the newest version. Same thing about the new QUAD engine – next 1.08BETA is faster and more reliable and at the same time fully compatible with 1.07BETA...

[nimdamsk]

I appreciate hard work of encode and other people on the scene. They move it forward surely. But please answer, why do so many people write so much compressors nowadays?
Are they strongest at compression? No. Undoubtedly leaders are PAQ-like algorithms with CM and neuro-tricks. As for WinRK, IMHO it's main power are good filters, not compression algoritm, may be i am wrong.
I don't want to criticize anyone, but here is quick'n'dirty test i've just made. I compressed full installed NERO folder with different archivers.

Folders 151
Files 1923
Files size 410,775,058

7z │ 144,899 K - 154 sec - 16mb dict, 16 word size
rar │ 149,009 K - 173 sec - maximum
uha│ 152,323 K - 116 sec - 32mb dict, mz
SBC│ 156,479 K - 184 sec - b9
pim│ 177,427 K - 359 sec - default
zip│ 192,518 K - 83 sec - maximum in WinRAR

So, no comments. Yes, i am not testing guru, but nevertheless, it's a typical situation. I also understand that 7-zip is been developed several years (earlier as BIX, 777 and UFA), and that you can further enchance your programs.

I think that Ilia improved his algoritms greatly and discovered some new ones. Can you turn from tuning heavy CM-like algoritms to fast BWT and LZxx? I'd like to see opensource THOR competitor. You can also tune LZX algorithm, it has great decompression speed.

[encode]

The testing results are heavily depends on data, since each algorithm has a strong and a weak sides. I hope you and others understand that.

Can you turn from tuning heavy CM-like algoritms to fast BWT and LZxx?

Well, I have my own line of algorithms, own area of interests. I dont like BWT. The latest QUAD, which Im working now, is LZ-based.

Id like to see opensource THOR competitor.

IMHO, THOR is too fast with too weak compression. For fast compression/decompression coders nowadays Ill recommend something LZH-based, like Deflate (ZIP) and RAR in Fast/Fastest mode.

You can also tune LZX algorithm, it has great decompression speed.

LZX is good, but it already invented and well implemented – no reason to make another clone. Again, I have my own ideas and even if my programs will be somewhere in the middle, its okay.

[nimdamsk]

IMHO, THOR is too fast with too weak compression. For fast compression/decompression coders nowadays Ill recommend something LZH-based, like Deflate (ZIP) and RAR in Fast/Fastest mode.

May be. Lets check I archived Nero folder into zero-compressed 7z archive.

ex │ 179,588 K - 26 sec - thor ex
zip│ 191,109 K - 97 sec - WinRAR max compression
zip│ 200,760 K - 34 sec - WinRAR max speed
│ 401,171 K - original

And, for reference

pim│ 157,816 K - 436 sec

Again, its a typical situation, ive made such test many times.

And NERO archive is a good example. Its biggest 200 files (361,093,668 bytes from full 410,775,058 bytes) are:
258,969,380 bytes in 152 files - PE 32 executables
58,245,952 (37) - almost uncompressible 7z, img, jpg, mpg, nct, png, VDB, zip
40,740,448 bytes in 8 files - bmp
2,789,728 bytes in 2 files - misc. binary
There are also 1723 small (~30 kb each) files

Can you tune LZSS? Isnt it interesting to make fastest practical archiver? I know you can do it

[encode]

Dont forget, THOR has a symmetric compression (except with exx mode) - the decompression speed is the same or even slower compared to the compression.

Isnt it interesting to make fastest practical archiver?

Archiver is the crazy thing. PIM is a pretty simple archiver, but source code is already too complex - its HUGE. So, actually, I have no idea for future improvements. Probably, the latest update with new EXE-filter and QUAD deompressor is the final essential update of this archiver.

Someday, I will have no spare time for data compression. Again, probably QUAD is the last thing. So, these days, Im hardly working on QUAD, making code faster and more reliable.

LZSS? I started a data compression trip from simple LZSS compressor. Somewhere in the middle I turned to LZP, ROLZ and PPM...

A very sentimental and philosophic topic...

[nimdamsk]

Dont forget, THOR has a symmetric compression (except with exx mode) - the decompression speed is the same or even slower compared to the compression.

With thors speed it doesnt matter

LZSS? I started a data compression trip from simple LZSS compressor. Somewhere in the middle I turned to LZP, ROLZ and PPM...

I know that compression is your hobby only. Sadly, as i see, you dont want to make practical archiver, being interested in inventing new compression chemes. So, there are two ways - to sit and wait that one day someone (I. Pavlov?) will make 7-zip perfect or to make it myself. The problem is i cant But it can ve solved anyway.

Thank you again for your work.

[Bulat Ziganshin]

about FreeArc:
1) my program homepage now is http://www.haskell.org/bz
2) it divides all the files compressed into the text and binary groups and uses lzma for binary and ppmd for text files. also, i've added a bit of preprocessing techniques, you can see the whole compression algorithm by using --display option

in *my* own tests, program works better than 7-zip, and is close to the uharc -mx. unfortunately, winrk craches in most of my tests (probably because i typically compress thousands of files), but i agree that winrk has best compression, on cost of speed

mmx/sse can't be used in compression algorithms. multithreading, otoh, is the hottest topic now. i think it will be especially useful for improving ppmd speed because ppmd actually splits all the data in 6-7 mb blocks (with optimum settings), which are compressed independently. moreover, ppmd don't improve its compression when more than, say, 200 mb are available

for best comression one should use -m5..-m8 options (depending on memory available, -m5 - 256megs, -m8 - 2gigs)

alternatively, you can use -m5p..-m8p if external ppmonstr.exe is available on your system (it will be required for decompression too!)

about MM compression - it's a myth. MM compression can be used only for *decompressed* audio/picture data - are you really have this? it can't compress mp3/avi/jpg files

[Bulat Ziganshin]

about THOR: it uses lzh algorithm which is of course assymmetric. it uses large dictionary with a short search chain, so it works better than 32kb-dictionary zip or rar -m1. it's my beloving algorithm too. if his author will not release his algorithm under GPL license, once i will try to develop something like this

meanwhile, try freearc with -m2 option. in my tests, it beats uharc -mz

-m3 beats rar. -m4 beats 7-zip

[encode]

FreeArc is very promising, but has no GUI...

mmx/sse cant be used in compression algorithms.

MMX works fine with PAQ8. However, with QUAD MMX/SSE not really works. With some experimental versions of QUAD I parallelize many loops using SSE - this gives no speed improvements - speed becomes even slower.

about MM compression - its a myth. MM compression can be used only for *decompressed* audio/picture data - are you really have this? it cant compress mp3/avi/jpg files

Its not a myth. BMP and WAV files are both contains multimedia data. Some users keep audio files in WAV (PCM) files and pictures in BMP (lossless). Your program just hasnt MM compression. It is worth to have at least simple delta filter for audio files - isnt it? Furthermore, many latest games have WAV file in their distribution. Why not to use MM compression? mp3/avi/jpg - are also multimedia files, but theyre already compressed - why compress twice??

[encode]

about THOR: it uses lzh algorithm which is of course assymmetric.

Nope. Only with exx. With others THOR uses LZP+HUFFMAN. As author said.

[Black_Fox1]

Hello Bulat and welcome to the forums! Are you the one who developed Rep preprocessor?

[MM compression] cant compress mp3/avi/jpg files

mp3s could be precompressed to wav with MM applied to these then, couldnt they?

[Bulat Ziganshin]

>Nope. Only with exx. With others THOR uses LZP+HUFFMAN. As author said.

10x! i was in doubt whether one should use lzh or lzph for fast compression. now i will look for second alternative. are you know lzp preprocessor from grzip? it may be further improved to reach thor-like compression and speed

>Are you the one who developed Rep preprocessor?

yes. although it's very like rzip and lzp preprocessor, i only added a few ideas. but where you can hear about it?

>mp3s could be precompressed to wav with MM applied to these then, couldn't they?

if you mean that mp3s can be *decompressed* to wav - yes, but *lossless* MM compression will be worse than original lossy mp3 compression

are you ever tried to compress avi/mp3/jpg with a MM archivers - rar, ace, uharc, winrk? that results are you got?

[Bulat Ziganshin]

>FreeArc is very promising, but has no GUI...

it seems that hardcore compression users want 1) compression, 2) CLI features, and only 3) GUI. and for me developing GUI would be rather hard - i don't had delphi-like RAD tools

>MMX works fine with PAQ8.

but not in practical algorithms. users often believe that some over-advertised technology like MM compression or SSE8.5 can improve their programs i don't know places in my current algorithms where mmx/sse can be used at all

>It's not a myth. BMP and WAV files

we can ask this partiicular user whether he really has uncompressed media files 12 years ago arjz was second program that included MM compression in its arsenal. but any feature need a time to develop and i don't want to spend my resources to work hard on real MM algorithms nor want to include simple delta just to say that my program includes MM compression. i just ask users why they need this. alongside of game and raw photo compression, i don't heared practical reasons

ps: i'm really amazed how fast i got answers here! it seems that noone sleeps and all watch this forum 24h a day thank you, Ilia, for establishing such lovely place for communication!

[Black_Fox1]

yes. although its very like rzip and lzp preprocessor, i only added a few ideas. but where you can hear about it?

here and here

mp3s can be *decompressed* to wav - yes, but *lossless* MM compression will be worse than original lossy mp3 compression

Oh, I know this is completely impossible. I expressed my thoughts wrong... if MP3 is Huffman coded, then it can be partially decoded and coded with some stronger algorithm again (Soundslimmer, also StuffIt uses something similar for packing JPGs) - this could be an example of useful MM

[Bulat Ziganshin]

>Soundslimmer

of course, but it is not what is already implemented in other archivers. in particular, Soundslimmer was several months of work of very experienced programmers. i don't have enough skills to repeat their result. the same applies to jpeg compression. i guess that you know all these bells and whistles

but i'm open to ideas of improving compression without hard work. good open-source libs that i can add to the program or easy-to-implement ideas. now i has ecm and ability to run external compressors in my list

>here and here

thank you! i will ask Ed for the file that shows up REP problems

[encode]

By the way, why Haskell?

[Bulat Ziganshin]

you may aks better - why archiver? when i started to learn Haskell, i has not found better program to implement using it. with all compression algorithms preimlemented in C, i can develop only rather high-level code and it's the area where Haskell rocks

otherwise, i leaved this business 5 years ago and don't had plans to return to it

[encode]

I have just a little knowledge about Haskell. So, is it possible to compile FreeArc under Linux? I bet CL programs are much popular under *nix platforms.

[Bulat Ziganshin]

i never tried. but i just got suse and will try it. probably i will make arc unix-compatible due this month (and it will become best unix archiver because now p7-zip is best and my programs uses same lzma/ppmd with addiional tricks

btw, now it uses rep, lzp, dict and bcj preprocessors (7zip uses bcj2, otoh), supports automatic selection of compression algorithm on file extension (but one need to write all the textfile extensions in arc.groups files, otherwise files will be compresed as binary), provides advanced file sorting algorithm. so, it's a great shell for compression algorithms. i tried not to not develop any compression algorithms anymore, but sometime has written two small preprocessors, dict and rep, to solve rather easy problems that was not solved by anyone else

so i hope to further develop program in the direction of inclusion other's algorithms, adding small preprocessors and enhancing overall program usability

[Fallon]

FreeArc is very promising, but has no GUI...

it seems that hardcore compression users want 1) compression, 2) CLI features, and only 3) GUI. and for me developing GUI would be rather hard - i dont had delphi-like RAD tools

Okay, its hard. And if you program for hardcore compression users, thats better than for nobody!
But to reach a wider audience, good compression needs a GUI. There is no way around it.

Shortlist of general GUI requirements:
1 - a GUI should loose little time, as most compression jobs are just a necessary waste of time.
2 - a GUI has to be very(!) intuitive, meaning: as-few-clicks-as-possible to get to any feature, the shortest route to any command, in terms of time, should survive.
3 - features should be useful.
4 - for many features, the option to put/group them in profiles, maintains top intuitivity.
5 - integration with popular GUIs is ok, starting with OS rightclick menu.

i just got suse and will try it. probably i will make arc unix-compatible due this month (and it will become best unix archiver

Best of luck with further development.

[Bulat Ziganshin]

>But to reach a wider audience, good compression needs a GUI. There is no way around it.

i know. just want to mention that if i started with great gui, then implemented features, and then goes into implementing compression, it will be even worser

[JohanDeBock]

For me this is a nobrainer: 7-Zip, and it has yet to include real multimedia filters!

I hope Igor hasn't dropped the idea, It was once on his todo list.

[encode]

А вообще по поводу "Бест":
На вкус и цвет товарища нет!

About the "best" term:
[Russian proverb, partially untranslatable Russian folk]
There is no friend for the same taste and color!

[Bulat Ziganshin]

>For me this is a nobrainer: 7-Zip, and it has yet to include real multimedia filters!

are you mean here bmp/wav or jpg/mp3 compression?

[JohanDeBock]

are you mean here bmp/wav or jpg/mp3 compression?

Any uncompressed image or audio data.

[JohanDeBock]

Try 7-Zip, Quad, RAR, UHARC, WinRK ROLZ3 and CCM on this file:
http://www.mediafire.com/?2ddtddq3dtl

[Bulat Ziganshin]

it's better to say what it is the type of data and where such files are used

[JohanDeBock]

1.000.000 bytes of wav data without header:
http://www.mediafire.com/?1y3zj3j5dmm

680,106 wav.uha
680,915 wav.ccm
688,862 wav.paq8l
695,184 wav.sbc
696,073 wav.rar
803,265 wav.mcomp
893,424 wav.hook
929,062 wav.7z
938,003 wav.sqx
954,961 wav.quad
969,378 wav.zip
1,000,000 wav

Here you can clearly see UHARC's superior generic wave data compression.
For example Squeez performs really bad here because it checks on extension or header to perform it's specialized wav routines.