JPEG Compression Test [December 2009]

[Skymmer]

JPEG Compression Test

December 2009

OK, here is the JPEG compression test I've finally made. The idea is kinda simple - to see how different currently existing JPEG compressors behave comparing to each other and to measure some other aspects. I must say that I've met some problems during it, but let's go by the numbers.

Testbed and system.
I just took one photo session maded with SONY DSC-W50 camera. It consists of 295 JPEG files. All files have 2816x2112 or 2112x2816 resolution depending on orientation. Original overall size is 602 112 057 bytes.
All files are not Progressive, Huffman tables are not optimised.
I've started the test on these original files but then the problem with PreComp has appeared. I've reported it here. Then I just stripped all non-image data from JPEG files with PureJPEG tool and final TestBed size became 596 454 271 bytes. Fine, but then another problem has appeared. Now with PAQ8px_67. For some reasons it doesn't recognize such cleared files as JPEGs and uses default model instead. Re-running the test for the 3rd time was too much so PAQ8px have been tested on original (non-stripped) data and listed separately.

Test conducted on Win XP Pro SP2, AMD64 4000+ (Single Core).

Competitors and settings.

PackJPG v2.3c                           default \ -dev -s17 -c14 (showed as custom)
WinZIP v14.0 #8646 (console)            -ez
PreComp v0.4.0 (PackJPG v2.4 WIP4)      default
StuffiT v13.0.0.29                      --jpeg-no-thumbnails
StuffiT v14.0.0.15                      --jpeg-no-thumbnails
PAQ8px_67 (speed_optimised)             -6

The things that you're probably looking for.
Well, sorry for long introduction and let's go to the tasty part

                         size        comp.   decomp.     mem.       %
                     -----------     -----   -------   -------   -------
Original             596 454 271      ***      ***      *****    100.0 %
PackJPG              479 216 216      955      957      31 MB    80.34 %
PackJPG (custom)     478 209 609     1015     1014      31 MB    80.18 %
WinZIP               476 061 457      579      517      18 MB    79.82 %
PreComp              472 273 653     1107     1079      28 MB    79.18 %
StuffiT 13           456 667 561      887      906      14 MB    76.56 %
StuffiT 14           456 645 014      970      882      22 MB    76.56 %

Original             602 112 057      ***      ***      *****    100.0 %
PAQ8px               461 483 178    15877    15877     470 MB    76.64 %

The final mini-battle
Since its not very clear who is the winner I've conducted additional mini-test between StuffiT and PAQ8px_67 on 10 JPEG files.

PAQ8px_67       34 273 729
StuffiT 13      34 013 770

How progressive you are ?
Let's see what competitors support Proressive JPEGs.

PackJPG       Yes
WinZIP        No
PreComp       Yes
StuffiT       Yes
PAQ8px        No

Blah Blah Blya Blah
First of all it's a little bit sad to see the performance of PackJPG v2.3c but results of PreComp v0.4.0 (PackJPG v2.4 WIP4) are very promising. Though, there is noticeable speed sacrifice.
WinZIP surprised me a lot. Not very bad compression and relatively blazing speed. Further more WinZIP is partially asymmetric which is good but lack of support for Progressive JPEGs is bad. Considering that the WZ developers are little bit care about compatibility I think that situation will not change in near future. Can be wrong though.
StuffiT results are very good but it was expected. Surely they did a good job but overall company politics and style just killing me.
PAQ8px is very close to StuffiT but gee... too slow and too high memory requirements and don't forget about lack of support for Progressive JPEGs.
Also, problems I've met during testing bring some questions. Is it JPEG format so badly developed so its hard to include its full support into product or its just a lazy developers or maybe it was just a bad choice of test material from my side?
Also, I have suspicion that Power Archiver will have its JPEG compressor sooner or later so let's wait...
Anyway, I hope the test was usefull and thanks for reading!

EDIT: added notice that all tested files are not Progressive JPEGs and Huffman tables are not optimized.
EDIT2: new result for PackJPG added.

[schnaader]

Nice test!

PAQ8px is very close to StuffiT but gee... too slow and too high memory requirements and don't forget about lack of support for Progressive JPEGs.

You should try PAQ with -3, too. It should be a bit faster (about 33%), has moderate memory requirements (<64 MB) and ratio shouldn't drop that much.

Using Precomp -progonly before would improve ratio for compressors that don't support progressive JPGs (for PAQ, you could also give paq8o8pre a try), but it depends on how much progressive JPGs actually are in the testbed, and I'm not sure if the result will be better than StuffIt. Especially Precomp -progonly + WinZip could be interesting, as it still should be fast.

For Precomp, I can answer the question about problems with JPG files. Precomp has to detect JPG files before passing them on to PackJPG and it doesn't do a great job here for "exotic" images that doesn't use the usual JPG structures. Like you've also seen with PAQ, JPG detection is not as easy as it seems if you can't rely on the extension. There is no header you could use that says where the JPG stream starts and how long it is, but the file consists of different parts with small headers (often 2 bytes) that can occur in different orders.

[Shelwien]

Cool, thanks.
Can you also try extracting winzip's archive using this:
http://nishi.dreamhosters.com/u/dejpg.exe ?
I'm interested in its decoding speed vs winzip.
And also whether it would be able to correctly decode your files.

[Skymmer]

Everything went fine! Extracted files are identical to originals. The results are:

WinZIP     517 sec.
deJPG      464 sec.

Nice! 10.25% faster. IntelC magic?
Also, last strings from console:

"DSC01862.JPG": CRC matches (687C332A). mem: 2.2M
Unknown tag!
Files extracted: 295, crc errors: 0, decode errors: 0

[m^2]

Like you've also seen with PAQ, JPG detection is not as easy as it seems if you can't rely on the extension. There is no header you could use that says where the JPG stream starts and how long it is, but the file consists of different parts with small headers (often 2 bytes) that can occur in different orders.

Something you might not know:
there's another person who wrote such detection scheme before.
http://mark0.net/soft-bitmaprip-e.html

Possibly you could learn from each other.

[SvenBent]

Why testing on single core ?

precomp and packjpeg are easy to make some multi threaded batch files for decompression.


I would love to retry this test on my multicore machine.
or dos it possible to get the test set or does it contains very privat data

[Raymond_NGhM]

Skymmer wrote:
Competitors and settings.

PackJPG v2.3c                           default
WinZIP v14.0 #8646 (console)            -ez
PreComp v0.4.0 (PackJPG v2.4 WIP4)      default
StuffiT v13.0.0.29                      --jpeg-no-thumbnails
StuffiT v14.0.0.15                      --jpeg-no-thumbnails
PAQ8px_67 (speed_optimised)             -6

For packJPG,
1st, try with -d switch
2nd, use -d -s8 -c7 switches...

[Skymmer]

You should try PAQ with -3, too. It should be a bit faster (about 33%), has moderate memory requirements (<64 MB) and ratio shouldn't drop that much.

Nice idea, I'll give it a try tonight.

Using Precomp -progonly before would improve ratio for compressors that don't support progressive JPGs (for PAQ, you could also give paq8o8pre a try), but it depends on how much progressive JPGs actually are in the testbed, and I'm not sure if the result will be better than StuffIt. Especially Precomp -progonly + WinZip could be interesting, as it still should be fast.

Well, all files are not Progressive so I suppose there is no need to test paq8o8pre and Precomp -progonly + WinZip pair.

Why testing on single core ?

Well, because my CPU has only one core

I would love to retry this test on my multicore machine.
or dos it possible to get the test set or does it contains very privat data

I don't know if photos, where I'm half-naked jumping under the rain with bootle of moonshine, can be called private
Now seriously. Nothing special there. Just a photos of Stockholm maded by my sister when she was on some medical symposium there. But I'm pretty sure she'll not like the idea that her photos will be sent to somebody whom she even doesn't know. Hope you'll understand.

For packJPG,
1st, try with -d switch
2nd, use -d -s8 -c7 switches...

Fascinating! I've ran PackJPG with development switch and it showed some interesting command line switches. Not too much to play with but nice anyway. Thanks for tip, Raymond !

[Skymmer]

Small but pleasant update of result table have been added. Following the advise from Raymond_NGhM I've played with -s and -c values in PackJPG. More exactly speaking I performed some kind of brief bruteforcing and found good combination of switches which allowed to improve PackJPG's result for 1 MB. Not too much but anyway nice! It listed separately from default result. Please read the Competitors and Settings section for details.

[schnaader]

Small but pleasant update of result table have been added. Following the advise from Raymond_NGhM I've played with -s and -c values in PackJPG. More exactly speaking I performed some kind of brief bruteforcing and found good combination of switches which allowed to improve PackJPG's result for 1 MB. Not too much but anyway nice! It listed separately from default result. Please read the Competitors and Settings section for details.

By the way, just if you're interested about best settings without too much bruteforcing, there was some topic (in the old forum I think) about this, I'll have a look if I can find it. The results of my brute force on A10.jpg, however, that were discussed there, are still available here: http://schnaader.info/graph_complete.png

As you can see, the resulting graph is quite regular, so instead of bruteforcing 252*62 = 15,624 times, you can basically do a 2D binary search in the region (3..50),(2..64) which should need about log2(64)*log2(50) = 34 tries. You often won't get the optimal result this way, but one very close to it.