4x4 - multithreaded compressor

[Bulat Ziganshin]

4x4 is a long-awaited multi-threading compression shell around tornado, lzma and grzip compression libraries. enjoy!

http://www.haskell.org/bz/4x4ver01.zip

the most exciting fact about 4x4 is that it's just 200 lines long and was developed just in 1 day. Haskell is amazing for making multi-threaded apps!

[encode]

Funny name, alternatives are 2+2=5, 2x2, 1+1=3... As far as I understand 4x4 is from car world (Four-wheel drive) - like 4x4 Hummer...

[Bulat Ziganshin]

4x4 is AMD initiative for multicore multiprocessor systems for this app this may mean "4 cores busy compressing and decompressing data using 4 algorithms" but actually there are only 3 algorithms and decompression is single-threaded. there is a room for future versions

added: i forsget about most important cause - it allows program to be placed before 7-zip in alphabetic listings

[LovePimple]

Thanks Bulat!

[Bulat Ziganshin]

my quick scanning of test1 results shows that now in PF 4x4 replaced lzturbo occurences. it's not because it's any better, but probably because its settings turns out to be better for this particular test:

 
quicklz	1.4.0 quick2  619,945,569 21  00:11.980 5,836,800     29 00:08.096 5,832,704 
thor	0.96  thor e2 587,980,752 22  00:12.136	3,305,472     30 00:11.466 3,293,184 
tornado	0.3   tor -4  447,968,986 23  00:17.472	23,175,168    31 00:07.597 11,689,984 
4x4	0.1   4x4 6   375,149,040 24  00:52.104	304,578,560   28 00:09.968 38,682,624 
4x4	0.1   4x4 7   349,846,233 31  01:40.823	706,539,520   27 00:09.328 72,265,728 
4x4	0.1   4x4 8   340,677,122 41  02:10.791	1,414,393,856 37 00:22.869 139,313,152 
tornado	0.3   tor -7  334,050,397 66  01:02.322	111,325,184   31 00:08.970 69,476,352 
tornado	0.3   tor -8  313,161,387 94  01:31.166	212,226,048   32 00:08.346 136,712,192 
tornado	0.4a  tor -9  295,759,477 104 01:40.527	414,216,192   39 00:08.002 271,220,736 
tornado	0.4a  tor -10 279,273,808 152 02:29.838	817,741,824   32 00:07.644 540,184,576 
tornado	0.4a  tor -11 267,109,958 240 03:57.293	1,624,543,232 33 00:08.377 1,078,104,064

it will be great to find way to cut off last 3 seconds from 4x4 runtimes

unfortunately i had troubles sending program to testing. thanks to anonymous tester who done this work!

[encode]

(Bulat, please, don't post such *wide* results, like you see due to a forum bug the main table is widened due to such content. Or I will restrict the longest word (line)!)
EDIT: Fixed!

[Bulat Ziganshin]

4x4 version 0.2 - bigger, longer, uncut
* fast compression modes made even faster
* multi-threaded decompression
* fast & efficient text compression modes 1t..4t using grzip
* Win32 & Linux versions

On average, 4x4 is 12 times faster than lzturbo!!!

Below are detailed results from
http://www.metacompressor.com/uploads.aspx

file1 - big text 1,552,758,784 
 
                  mode  size          compression/decompression time 
    4x4     0.2   1t    354,958,849   24   29 
    lzturbo 0.92  -59   322,372,806   219  21 
    4x4     0.2   4t    311,810,707   41   36 
    4x4     0.2   9     300,944,001   117  20 
 
* 4x4 is 5 times faster than lzturbo!

file2 - windows 2008 server virtual disk 1,489,400,832 
 
                  mode  size          compression/decompression time 
    lzturbo 0.92  -59   352,723,630   1343 
    4x4     0.2   6     344,522,813   25   24 
    4x4     0.2   9     268,639,166   97   19 
 
* 4x4 is 53 times faster than lzturbo!!!

file3 - big apache log 2,072,612,014 
 
                  mode  size          compression/decompression time 
    lzturbo 0.92  -59   103,156,939   313 
    4x4     0.2   4t    80,941,237    24   28 
 
* 4x4 is 13 times faster than lzturbo!

file4 - enwik9 part wikipedia export 1,000,000,000 
 
                  mode  size          compression/decompression time 
    4x4     0.2   1t    233,368,003   14   18 
    lzturbo 0.92  -59   232,701,587   155  14 
    4x4     0.2   4t    208,787,642   34   26 
 
* 4x4 is 11 times faster than lzturbo!

file6 - source code linux, apache, mysql, php and firefox 729,287,225 
 
                  mode  size          compression/decompression time 
    lzturbo 0.92  -59   156,185,652   109 
    4x4     0.2   1t    151,456,761   14   12 
    4x4     0.2   9     151,252,453   51   9 
 
* 4x4 is 8 times faster than lzturbo!

[LovePimple]

Thanks Bulat!

[nimdamsk]

Hey! You spoil someone's business. Shame on you!

[Bulat Ziganshin]

finally, we got results for last file:

file7 - big db index 1,066,160,748 
 
                  mode  size          compression/decompression time 
    lzturbo 0.92  -59   406,358,090   4405 20 
    4x4     0.2   1t    390,897,470   35   34 
    4x4     0.2   9     359,413,561   145  13 
 
* 4x4 is 125 times faster than lzturbo! no comments :)

it seems that lzturbo optimal parsing is too slow on binary files to make it any usable, while on text files it easily outperformed by bwt/st4/ppmd algorithms. good for nothing

[Simon Berger]

Hmmmm. Didn?t you shake your head on reading those posts again later?

[Black_Fox1]

Much more logical than hiding those results from others
EDIT: I mean, it means something when you're more than 10x faster than world's fastest compression library, think about it.

[Bulat Ziganshin]

Simon, i realize that i don't like Hamid and remember reasons to do it. But are you remember why you don't like me?

on technical grounds, there are not so many multithreaded compressors ATM, so comparison of two best ones seems logical, are you not agree? you definitely forgot that just one month ago you criticized me in exactly opposite direction - why i don't write lzturbo competitor if i think that it is so easy