Results 1 to 13 of 13

Thread: 4x4 - multithreaded compressor

  1. #1
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,507
    Thanks
    742
    Thanked 665 Times in 359 Posts
    4x4 is a long-awaited multi-threading compression shell around tornado, lzma and grzip compression libraries. enjoy!

    http://www.haskell.org/bz/4x4ver01.zip

    the most exciting fact about 4x4 is that it's just 200 lines long and was developed just in 1 day. Haskell is amazing for making multi-threaded apps!

  2. #2
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Funny name, alternatives are 2+2=5, 2x2, 1+1=3... As far as I understand 4x4 is from car world (Four-wheel drive) - like 4x4 Hummer...

  3. #3
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,507
    Thanks
    742
    Thanked 665 Times in 359 Posts
    4x4 is AMD initiative for multicore multiprocessor systems for this app this may mean "4 cores busy compressing and decompressing data using 4 algorithms" but actually there are only 3 algorithms and decompression is single-threaded. there is a room for future versions

    added: i forsget about most important cause - it allows program to be placed before 7-zip in alphabetic listings

  4. #4
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Bulat!

  5. #5
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,507
    Thanks
    742
    Thanked 665 Times in 359 Posts
    my quick scanning of test1 results shows that now in PF 4x4 replaced lzturbo occurences. it's not because it's any better, but probably because its settings turns out to be better for this particular test:

    Code:
     
    quicklz	1.4.0 quick2  619,945,569 21  00:11.980 5,836,800     29 00:08.096 5,832,704 
    thor	0.96  thor e2 587,980,752 22  00:12.136	3,305,472     30 00:11.466 3,293,184 
    tornado	0.3   tor -4  447,968,986 23  00:17.472	23,175,168    31 00:07.597 11,689,984 
    4x4	0.1   4x4 6   375,149,040 24  00:52.104	304,578,560   28 00:09.968 38,682,624 
    4x4	0.1   4x4 7   349,846,233 31  01:40.823	706,539,520   27 00:09.328 72,265,728 
    4x4	0.1   4x4 8   340,677,122 41  02:10.791	1,414,393,856 37 00:22.869 139,313,152 
    tornado	0.3   tor -7  334,050,397 66  01:02.322	111,325,184   31 00:08.970 69,476,352 
    tornado	0.3   tor -8  313,161,387 94  01:31.166	212,226,048   32 00:08.346 136,712,192 
    tornado	0.4a  tor -9  295,759,477 104 01:40.527	414,216,192   39 00:08.002 271,220,736 
    tornado	0.4a  tor -10 279,273,808 152 02:29.838	817,741,824   32 00:07.644 540,184,576 
    tornado	0.4a  tor -11 267,109,958 240 03:57.293	1,624,543,232 33 00:08.377 1,078,104,064
    it will be great to find way to cut off last 3 seconds from 4x4 runtimes

    unfortunately i had troubles sending program to testing. thanks to anonymous tester who done this work!

  6. #6
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    (Bulat, please, don't post such *wide* results, like you see due to a forum bug the main table is widened due to such content. Or I will restrict the longest word (line)!)
    EDIT: Fixed!

  7. #7
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,507
    Thanks
    742
    Thanked 665 Times in 359 Posts
    4x4 version 0.2 - bigger, longer, uncut
    * fast compression modes made even faster
    * multi-threaded decompression
    * fast & efficient text compression modes 1t..4t using grzip
    * Win32 & Linux versions

    On average, 4x4 is 12 times faster than lzturbo!!!

    Below are detailed results from
    http://www.metacompressor.com/uploads.aspx

    Code:
    file1 - big text 1,552,758,784 
     
                      mode  size          compression/decompression time 
        4x4     0.2   1t    354,958,849   24   29 
        lzturbo 0.92  -59   322,372,806   219  21 
        4x4     0.2   4t    311,810,707   41   36 
        4x4     0.2   9     300,944,001   117  20 
     
    * 4x4 is 5 times faster than lzturbo!

    Code:
    file2 - windows 2008 server virtual disk 1,489,400,832 
     
                      mode  size          compression/decompression time 
        lzturbo 0.92  -59   352,723,630   1343 
        4x4     0.2   6     344,522,813   25   24 
        4x4     0.2   9     268,639,166   97   19 
     
    * 4x4 is 53 times faster than lzturbo!!!

    Code:
    file3 - big apache log 2,072,612,014 
     
                      mode  size          compression/decompression time 
        lzturbo 0.92  -59   103,156,939   313 
        4x4     0.2   4t    80,941,237    24   28 
     
    * 4x4 is 13 times faster than lzturbo!

    Code:
    file4 - enwik9 part wikipedia export 1,000,000,000 
     
                      mode  size          compression/decompression time 
        4x4     0.2   1t    233,368,003   14   18 
        lzturbo 0.92  -59   232,701,587   155  14 
        4x4     0.2   4t    208,787,642   34   26 
     
    * 4x4 is 11 times faster than lzturbo!

    Code:
    file6 - source code linux, apache, mysql, php and firefox 729,287,225 
     
                      mode  size          compression/decompression time 
        lzturbo 0.92  -59   156,185,652   109 
        4x4     0.2   1t    151,456,761   14   12 
        4x4     0.2   9     151,252,453   51   9 
     
    * 4x4 is 8 times faster than lzturbo!

  8. #8
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Bulat!

  9. #9
    Member
    Join Date
    Jan 2007
    Location
    Moscow
    Posts
    239
    Thanks
    0
    Thanked 3 Times in 1 Post
    Hey! You spoil someone's business. Shame on you!

  10. #10
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,507
    Thanks
    742
    Thanked 665 Times in 359 Posts
    finally, we got results for last file:

    Code:
    file7 - big db index 1,066,160,748 
     
                      mode  size          compression/decompression time 
        lzturbo 0.92  -59   406,358,090   4405 20 
        4x4     0.2   1t    390,897,470   35   34 
        4x4     0.2   9     359,413,561   145  13 
     
    * 4x4 is 125 times faster than lzturbo! no comments :)
    it seems that lzturbo optimal parsing is too slow on binary files to make it any usable, while on text files it easily outperformed by bwt/st4/ppmd algorithms. good for nothing

  11. #11
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Hmmmm. Didn?t you shake your head on reading those posts again later?

  12. #12
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Much more logical than hiding those results from others
    EDIT: I mean, it means something when you're more than 10x faster than world's fastest compression library, think about it.

  13. #13
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,507
    Thanks
    742
    Thanked 665 Times in 359 Posts
    Simon, i realize that i don't like Hamid and remember reasons to do it. But are you remember why you don't like me?

    on technical grounds, there are not so many multithreaded compressors ATM, so comparison of two best ones seems logical, are you not agree? you definitely forgot that just one month ago you criticized me in exactly opposite direction - why i don't write lzturbo competitor if i think that it is so easy

Similar Threads

  1. FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)
    By Bulat Ziganshin in forum Data Compression
    Replies: 554
    Last Post: 26th September 2018, 03:41
  2. 4x4 bug?
    By m^2 in forum Data Compression
    Replies: 11
    Last Post: 15th November 2008, 20:25
  3. CCM(x) multithreaded ?
    By SvenBent in forum Forum Archive
    Replies: 2
    Last Post: 15th September 2007, 16:29

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •