Squeeze Chart

[Stephan Busch]

Such a standard container format with API support would be really nice since it could be used for testing commandline compressors as if they were
archivers. As a matter of fact, new ideas come too fast - so they are often
transformed into code and published as plain experimental file to file compressors - which often lacks wildcard support. But damn, I love that experimental compressors as those are visions for the future - apart from that, most of them today are good enough to compensate wildcard absence
by strong algorithm and advanced analyzing of TAR content.

Greetings to Erfurt

Stephan

[IsName]

Hm, i think that "external compressor" feature in freearc is enough stable for that

[Intrinsic]

I currently use FA for single file compressors, just setup your externals and methods in the ini and you're done Just saves me the hassle of having to use QFC or TAR to package stuff up.

[toffer]

Hi!

Thanks Stephan.

Well, but you miss one important point - single file compression still stays single file compression. You cannot use the redundancy between two files of identical types (which is often called "solid archive"). With a general interface to streaming compressors this can be accomplished - thus resulting in higher compression. The archiver interface only has to order the files by type and tell the coder to reset its model/state/whatever in between two different types.

[pat357]

single file compression. You cannot use the redundancy between two files of identical types (which is often called "solid archive"). With a general interface to streaming compressors this can be accomplished - thus resulting in higher compression. The archiver interface only has to order the files by type and tell the coder to reset its model/state/whatever in between two different types.

One could also use FA for this :

arc a -mcmm -se archive files

Will first group the files by extension and create a separate CMM per extension and putting all the CMM compressed files in one arc-archive.
Works great !

[toffer]

Sorry, i lack FA experience, my fault.

But i still think that a general C/C++ interface would be better, since otherwise you've got the overhead of duplicate initialisation, duplicate headers,... and stuff like that. Still nice to know that FA does it

BTW: So there's a FA integration for CMM4?

[pat357]

Sorry, i lack FA experience, my fault.

But i still think that a general C/C++ interface would be better, since you've otherwise you've got the overhead of duplicate initialisation, duplicate headers,... and stuff like that. Still nice to know that FA does it

BTW: So there's a FA integration for CMM4?

Not by default (ie not in the FA distro), but it's very easy to add CMM in the ARC.INI.

[Stephan Busch]

Dear readers,

after over a month of hibernation I have managed to put the updated main chart online. This update includes hundreds of new results from the waveform and bitmap testsets (mostly in deflate class) but also some new entries such as WinZip 12, PAQ8p1, WinRK 3.1.2 (corrected waveform decompression), NanoZip 0.04, PowerArchiver 11 (with optimized 7z compression), BIT 0.6 and QPRESS 0.11.

Greetz,

Stephan

[Nania Francesco]

Thanks ! Please in the next UPDATE , if possible , add BZP !

[LovePimple]

Thanks Stephan!

[osmanturan]

Thanks a lot!

Stephan, there are some small mistakes in your chart.

- CMM4 is not a LZP+Context Mixing. LZP used in old versions as a preprocessor in CMM. In CMM4 there is a match model which act as LZP. But, this doesn't mean LZP+Context Mixing. It should be "Context Mixing".

- BIT 0.2 and 0.6 description should be "Light-Weight Context Mixing" or "LWCX" or simply "Context Mixing". LWCX stands for Light-Weight Context Mixing (and it the description already includes SSE stage. So, LWCX+SSE is a wrong description). I choosed this name because it doesn't use heavy context mixing like PAQ series do. Also, BIT 0.1 can be just "ROLZ". Because, it's main algorithm was ROLZ (actually I don't remember the remaining parts, because I played with it a lot ).

- SR3a description should be "Symbol Ranking + Filters".

- WinRK in maximum mode, it only uses PWCM. Not use ROLZ-3.

- I think, NanoZip CM mode cannot be classified as FCM (fast context mixing). Because, it's speed worser than CCMX, CMM4 and even BIT. It should be "Context Mixing + Filters"

Note: If you think there is a mistake in my description, please let me know.

[Stephan Busch]

Dear Osman,

how are you? I hope everything is fine.
Thank you very much for telling the issues which will be fixed with next release.

WinRK in maximum mode of course doesn't use ROLZ3, but I used ROLZ3 for the CDImage set, because it compresses better using ROLZ3 than PWCM.
That's why both algo's are mentioned.

When it comes to fast content mixing, I compare it to the slow content mixing of PAQ6/7/8 family. CMM, CCM, NanoZip and your BIT are a lot faster than the slow content mixing - therefore I prefer to call them fast content mixers. If you agree, I would change BIT algorithm description to fast content mixing.

Yours,

Stephan

[osmanturan]

how are you? I hope everything is fine.
Thank you very much for telling the issues which will be fixed with next release.

Thanks Stephan. I'm fine. I hope, you are ok too. I'm very busy with four overlapped jobs + an English exam on 12th October So, I cannot make some serious update for BIT.

WinRK in maximum mode of course doesn't use ROLZ3, but I used ROLZ3 for the CDImage set, because it compresses better using ROLZ3 than PWCM. That's why both algo's are mentioned.

Thanks for detailed description. BTW, I wonder how ROLZ-3 outperform PWCM mode on audio

When it comes to fast content mixing, I compare it to the slow content mixing of PAQ6/7/8 family. CMM, CCM, NanoZip and your BIT are a lot faster than the slow content mixing - therefore I prefer to call them fast content mixers.

If we talk about speed, we cannot classify both CCMX, CMM etc. with PAQ series in same classification. So, you are right. It should be fast context mixing for CMM4, CCM(X), NanoZip and BIT. But, there are some small context mixer coder for experiment like M1, FCM etc. How should we call them? I think, we cannot classify CCMX and M1 in same classification.

If you agree, I would change BIT algorithm description to fast content mixing.

Yes. It would be nice. Thanks.

[Nania Francesco]

Stephan The next Squeeze Chart 2009 , when you will be released?

[Stephan Busch]

Hi folks,

finally the SqueezeChart has been updated.
I hope you enjoy reading..
If you find any typos just let me know


Yours,

Stephan

[nanoflooder]

Great job done I can't believe in the amount of time it took. That's just amazing.
How about paq8p1-exp04?

[encode]

Hi folks,

finally the SqueezeChart has been updated.
I hope you enjoy reading..
If you find any typos just let me know


Yours,

Stephan

It's great! Thank you! BTW, can you please fix the link to my forum at your main page?

[Stephan Busch]

hey nanoflooder,

can you please provide a link for paq8p1-exp04?
I could only find paq8p1-exp00 and paq8p1-exp03 in this forum
and those did not compress properly.
From which date are the source and exe?

the archive _paq8p1_exp00.zip contains _paq8p1.exe from 30th september
the archive paq8p1_exp03.zip contains paq8p1.exe from 31th of august..

which is the latest one to test?

[nanoflooder]

the archive _paq8p1_exp00.zip contains _paq8p1.exe from 30th september

Oh, it's actually called ext00 indeed... This was the latest build of them all, as you can see by the date, hence why I've called it exp04.

I could only find paq8p1-exp00 and paq8p1-exp03 in this forum
and those did not compress properly.

That's bad... You don't have the logs with the failure, do you?..

[m^2]

exp04 is from october.
I can't find the link, sb. from this forum showed me it, but I have the archive.

[Stephan Busch]

I'll just give the paq8p1_speed_optimised dated 30th of september a rerun.
I only remember having the exp03 tested which failed compressing the driver.tar.. This is why this PAQ8p1(exp03) is at the red bottom of the chart.
if exp04 which is named exp00 is the latest build there may be a chance everything works out fine.. and the WinRK is beaten and this PAQ is master of 2008.. There was a lot of marvellous work done with this builds and I hope it runs well over the driver.tar (just started compression)

I'll keep you updated

[Intrinsic]

Cheers for the update Stephan!

[osmanturan]

Thanks a lot!

[Nania Francesco]

THANKS STEPHAN ! PLEASE TEST MY BZP !

[Stephan Busch]

so far PAQ8p1 compressed successfully the driver.tar testset,
not all testsets have been compressed now, but according to my estimations
PAQ8p1 will achieve 2.068.000.000 bytes which is not better than WinRK.
It was not able to detect JPEG inside the mobile.tar testset which decreased
compression by 20 MB. Further observations showed:

* detect exe seems ok; on crossplattform.tar it seems correct (if with exe only win32/win64 pe executables are seen and not linux or macOSx exe
but on 3DGame.tar there are 2.957.824 bytes in .dll and 823.296 bytes in .exe files but PAQ sees just 2.535.999 bytes 'exe'
and on driver.tar we have:
.ax = 988.672 bytes
.dll = 109.225.640 bytes
.exe = 9.482.013 bytes
.drv = 308.736 bytes
.sys = 55.226.735 bytes
reported are just 68.275.842 bytes 'exe'
*detect 8bit/mono wav in modules.tar does not work
*cdimage.tar - 37 MB wav stereo 16bit found (I have to check if there is really a wav hidden in the .dat file)
* PPM/PGM/BMP detection is perfect

However, this is marvellous work and I am testing mobile testset again -
this time in a no-compression rar archive because I think the JPEG detection
may have failed because of weird .tar format and non-text symbols in filenames of some JPEG. The detection failed also on previous versions
and if it works with .rar container, it will be ranked #1 of 2008 pushing current master (WinRK) on 2nd position.
I really love the WAV and PPM/PGM detection and compression power.
If someone wants the complete logs, just let me know.

[Stephan Busch]

so far, only the freedb.tar needs to be compressed.. all other sets are done.
The JPEG of the mobile testset could not be detected regardless of the container format (no-compression tar, rar, 7z..) Those JPEG follw directly the .JAR archives - can this cause detection failure?

Good news to you guys: PAQ8p1 beats WinRK.. it compresses 3 MB better so
far and therefore is the new winner of 2008 !!

[Simon Berger]

Mhm... I think thats not really fair to search for better environments for some archivers (if they can't provide it internally).

One comment to your Benchmark and Uharc. I tested it really much some time ago and got sometimes really much better results in disabling media mode (-mm-). Especially on the compressed data you should get better speed but I also sometimes saw better compression.

[Stephan Busch]

Hi Simon,

btw.. all PAQ results are done with same conditions.. TARed..
it was just a try to figure out if the tar format was to blame for failing JPEG detection

so no need to worry..

[Stephan Busch]

The SqueezeChart has a new guestbook which is in english now
and allows comments, pictures and avatars..

The link to this forum has been updated.

[encode]

The SqueezeChart has a new guestbook which is in english now
and allows comments, pictures and avatars..

The link to this forum has been updated.

Thanks a lot!