The 1MB Compression Benchmark

[comp1]

Good evening everyone!

This benchmark is what the "DOS Archiver Benchmark" originally was supposed to be but then changed.

This is just a fun benchmark to see how much we can compress a file with the following 2 requirements:

1) The program can NOT exceed 1,999 K in Windows Task Manger during compression.
2) The compression program must be a Win32 or Win64 executable.

The file to be compressed is as follows:

- Microsoft Office folder "Office12"
- Size: 555 MB (582,474,448 bytes)
- Total Files: 2,054

The folder is archived in a single .shar file using Shelwien's SHAR archiver program with the following command:

shar a Office12.shar Office12\

And now for the results:

UPDATES
12/19/14 - Updated UFA with "-m9" switch
12/18/14 - Added UFA 0.04b1
12/18/14 - Added ARJZ 0.15
12/18/14 - Updated RAR with "-mm" switch
12/18/14 - Added IMP 1.12
12/18/14 - Added ASHFORD
12/17/14 - Added LZOP
12/17/14 - Added P5
12/17/14 - Added SNAPPY 1.1.1.8

SIZE        | PROGRAM/SWITCHES
==============================================
361,017,573 | SNAPPY 1.1.1.8
339,895,453 | FCM1
323,422,347 | P5
316,368,134 | LZARI
311,091,351 | LZOP 1.03 -9
304,518,765 | ASHFORD 4
295,034,882 | GZIP 1.3.5 -9
294,847,400 | HA 0.999b a12
292,049,003 | ARJZ 0.15 -mp9
289,085,400 | ICT Universal Coder 2.1 em1
287,445,924 | JAR32 1.02 -m3
285,495,858 | QUANTUM 0.97 -c7 -t16
285,197,211 | ACE32 1.2 -m5 -d64
283,091,108 | CABARC -m LZX:15
282,180,711 | RAR 2.00 -m5 -md128 -mm
281,889,329 | IMP 1.12 -1 -m3 -mm -u200
275,802,821 | PPMs J -o6
265,063,803 | UFA 0.04b1 -mx -m9 -md64

If you'd like to suggest or submit a program to be tested, please post or send me a PM. For submissions, please only submit Win32 or Win64 executables as I will not bother trying to compile sources.

I will continue to update this first post when new file compressors are found (by me or by others) that meet the requirements for this benchmark.

IMPORTANT NOTE: Please remember, this benchmark has absolutely no practical purpose--it is just for fun.

[Matt Mahoney]

If you have the patience you might want to try:

zpaq add office12.zpaq office12.shar -method 57 -threads 1
zpaq extract office12.zpaq office12.shar -to office12.out -threads 1
fc/b office12.shar office12.out

Or compress the folder instead of the shar file. It will probably compress better that way.

Get zpaq.exe or zpaq64.exe at http://mattmahoney.net/dc/zpaq.html

[Cyan]

lz4 -9 -B4D

triggers the 64KB block mode,
should remain below memory limit.

Pre-compiled windows binary

[comp1]

@Cyan

Ahhh so close!!!



@Matt



And it was still going up...

In regards to compressing the folder as opposed to one large file: The problem is that in some cases (like with ACE32), compressing a folder of 2,000+ files increases the memory usage past the limit. Also, some programs are not archivers but only compressors. I wanted to keep this competition as fair and consistent as possible.

[Cyan]

Pretty strange
I made several tests, and consistently received 1884KB as peak memory usage
(which is already too much, and shows that the underlying allocation practice is a bit too "generous").
It would be interesting to understand why there is such a difference between both systems.

In the meantime, it's also possible to directly compile from the open source code.
Memory usage should be much more frugal on this version.

[comp1]

Pretty strange
I made several tests, and consistently received 1884KB as peak memory usage
(which is already too much, and shows that the underlying allocation practice is a bit too "generous").
It would be interesting to understand why there is such a difference between both systems.

In the meantime, it's also possible to directly compile from the open source code.
Memory usage should be much more frugal on this version.

That is quite a difference...Interesting.

Well, I may look at compiling the source sometime but my plan was not to mess with compiling source code for this benchmark.

Thanks.

[Matt Mahoney]

Oh, I misread what you said. I thought you said 1,999 MB

[just a worm]

Well, I may look at compiling the source sometime but my plan was not to mess with compiling source code for this benchmark.

That's an attitute I sympathize with. Why should every user compile a program instead of the developer compiles it once? The developer usually knows the best how to compile his program and has all the required external libraries etc. But some people still distribute the source code only.