Page 1 of 2 12 LastLast
Results 1 to 30 of 51

Thread: LZTURBO 0.0.1 - Experimental version

  1. #1
    Member
    Join Date
    Aug 2007
    Posts
    30
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hi,
    lzturbo is an lz77 based multithreaded and linear scalable (in multicore systems) file compression program using a fast state-of-the-art optimal parsing (option -49) and a fast arithmetic coder with very fast decompression.
    Please compare with quicklz (option -11) or other compression-programs (option -49).

    There is no compressor that compresses better with faster decompression (option -49).

    download (at this time, only linux version):
    http://www.consultant-berater.de/lzturbo-0.0.1.tar .gz

    usage:
    lzturbo -1X input output
    lzturbo -2X input output
    lzturbo -4X input output
    X=1..9

    ex. compression:
    lzturbo -49 enwik9 enwik9.lz

    decompression:
    lzturbo -d enwik9.lz enwik9.lzd

    Enjoy!

  2. #2
    Member
    Join Date
    Jan 2007
    Location
    Moscow
    Posts
    239
    Thanks
    0
    Thanked 3 Times in 1 Post
    Thanks!
    Waiting for Win32 version

  3. #3
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by nimdamsk
    Waiting for Win32 version
    Me too
    Seems to be interesting!
    License? Source? (Linux does not necessarily mean GPL...)

    Best regards!

  4. #4
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by donotdisturb
    There is no compressor that compresses better with faster decompression (option -49).
    I doubt it. A revolution form noname? A bluff!

  5. #5
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by donotdisturb
    There is no compressor that compresses better with faster decompression (option -49).
    ... doesnt mean much at all, _except_ that -> for the decompression time LZTURBO 0.0.1 needs there is no other program with better compressing.
    So _I_ read -> decompression time is quick. Not more, but even not less!
    Cmon, give him a chance
    Im waiting for first results.

    Best regards!

  6. #6
    Member
    Join Date
    Jan 2007
    Location
    Moscow
    Posts
    239
    Thanks
    0
    Thanked 3 Times in 1 Post
    Who knew ccm and slug author several months ago?

  7. #7
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by nimdamsk
    Who knew ccm and slug author several months ago?
    And who wants to miss them now...?
    Still waiting for results.

    Best regards!

  8. #8
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by nimdamsk
    Who knew ccm and slug author several months ago?
    Unique case... By the way, looks like Chris waiting in shadows. Hes disappeared. As just as DARKs author! -- catch me if you can, you know...

  9. #9
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by encode
    -- catch me if you can, you know...
    Yeah, sometimes author play "hit and run"

    Best regards!

  10. #10
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Thanks donotdisturb!

    Could someone provide any test numbers/results?

  11. #11
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    I tried testing under Ubuntu but could not get it to work.
    Perhaps the problem is I can't link a 64-bit shared library to a 32-bit executable? I might get it to work with source code, or a Win32 exe.

    matt@matt-desktop:~/Desktop$ ./lzturbo
    ./lzturbo: error while loading shared libraries: libmagic.so.1: cannot open shared object file: No such file or directory

    matt@matt-desktop:~/Desktop$ file /usr/lib/libmagic.so.1
    /usr/lib/libmagic.so.1: symbolic link to 'libmagic.so.1.0.0'

    matt@matt-desktop:~/Desktop$ file /usr/lib/libmagic.so.1.0.0
    /usr/lib/libmagic.so.1.0.0: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), stripped

    matt@matt-desktop:~/Desktop$ file lzturbo
    lzturbo: ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), statically linked, stripped

    matt@matt-desktop:~/Desktop$ uname -a
    Linux matt-desktop 2.6.15-28-amd64-generic #1 SMP PREEMPT Thu May 10 09:46:40 UTC 2007 x86_64 GNU/Linux

  12. #12
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Matt Mahoney
    ./lzturbo: error while loading shared libraries: libmagic.so.1: cannot open shared object file: No such file or directory
    Same thing with Slax LiveCD...

  13. #13
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,507
    Thanks
    742
    Thanked 665 Times in 359 Posts
    Quote Originally Posted by encode
    I doubt it. A revolution form noname? A bluff!
    there are only two lz77 compressors with optimal parsing - cab and lzma. cab has very fast decompression but pretty simple model and only 2mb dictionary. otoh, lzma has sophisticated model which leads to several times slower decompression. i guess that lzturbo lies between these two programs in terms of both speed and compression

    imho, it should be possible to construct such algorithm from tornado sources (which provides lzari model) and lzma sources (which provides optimal parser)

    many years ago i "reimplemnted" arj algorithm by just combining zip match finder with ar002 model

    on my box (fedora 7/32 bit) i was able to run 1* and 2* models, but not 4*. -11 is only 2x faster than gzip -1 so probably it doesnt have any chances against quicklz

  14. #14
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Okay, some first digits (-49 mode):

    world95.txt: 581,517 bytes

    fp.log: 794,437 bytes

    ENWIK8: 26,678,709 bytes

    calgary.tar: 876,920 bytes

    It worked on Fedora 7 KDE Live. Compression is slow even on my machine, sometimes lzturbo even frozes... Decompression is fast indeed.


  15. #15
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    so here we are -> discussing facts!

    Best regards!

  16. #16
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Yet another results (-49):

    A10.jpg: 839,822 bytes
    acrord32.exe: 1,429,364 bytes
    english.dic: 792,403 bytes
    FlashMX.pdf: 3,741,354 bytes
    fp.log: 794,437 bytes
    mso97.dll: 1,827,499 bytes
    ohs.doc: 838,887 bytes
    rafale.bmp: 1,026,827 bytes
    vcfiu.hlp: 685,044 bytes
    world95.txt: 581,517 bytes
    Totally: 12,557,154 bytes

    pht.psd: stack overflow. Aborted

    pak0.pak: frozes

    PariahInterface.utx: frozes

    It uses up to 764 MB for compression. On some files it either crashes or freezes. In other words, it's buggy...

  17. #17
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    It's interesting to see the program's details (algo/implementation details). Also waiting for WIN32 release. (Maybe all bugs caused by Linux incompatibility...)

  18. #18
    Member
    Join Date
    Aug 2007
    Posts
    30
    Thanks
    0
    Thanked 0 Times in 0 Posts
    NEW:
    download linux version (with static-linked libmagic +libpthread)
    +win32-exe (win32-version tested only on linux!).

    http://www.consultant-berater.de/lzturbo-0.0.1.tar .gz

    Please read the file lzturbo.txt.
    Please use the linux version when possible.

    Enjoy!

  19. #19
    Member
    Join Date
    Aug 2007
    Posts
    30
    Thanks
    0
    Thanked 0 Times in 0 Posts

  20. #20
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Thanks donotdisturb!

    results on my testset (-49 parameter):
    (my 6 755 163 bytes PSD file produced stack overflow error and wasn't counted to stats)

    original size: 23 554 031
    compressed:13 227 143
    comp speed: 729 kB/s
    dcmp speed: 3 795 kB/s

  21. #21
    Programmer
    Join Date
    May 2008
    Location
    denmark
    Posts
    94
    Thanks
    0
    Thanked 2 Times in 2 Posts
    Hi donotdisturb,

    How about adding a -mem flag for benchmarking? Feel free to use the code from the quicklz demo

    Lasse

  22. #22
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks donotdisturb!

  23. #23
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    It works in Windows XP on enwik8/9 with one minor problem. The output files are read-only and have to be changed with "attrib -r" before you can modify or delete them. I didn't test the Linux version.
    http://cs.fit.edu/~mmahoney/compression/text.html# 2333

    It also beats lzpm for decompression speed and size.

  24. #24
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quick test with Win32 version...

    Test machine: AMD Sempron 2400+, WinXP SP2


    ENWIK8 (-11)
    Compression Time: 3.045 Seconds
    Compressed Size: 55,397,274 bytes

    ENWIK8 (-21)
    Compression Time: 3.298 Seconds
    Compressed Size: 52,727,086 bytes

    ENWIK8 (-22)
    Compression Time: 6.505 Seconds
    Compressed Size: 46,376,388 bytes

    ENWIK8 (-23)
    Compression Time: 13.229 Seconds
    Compressed Size: 39,822,079 bytes

    ENWIK8 (-24)
    Compression Time: 86.941 Seconds
    Compressed Size: 35,169,367 bytes

    ENWIK8 (-49)
    Compression Time: 312.522 Seconds
    Compressed Size: 26,678,709 bytes

    ENWIK8 (-19)
    Compression Time: 541.577 Seconds
    Compressed Size: 36,892,158 bytes

    ENWIK8 (-29)
    Compression Time: 542.872 Seconds
    Compressed Size: 33,883,186 bytes

  25. #25
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Matt Mahoney
    It also beats lzpm for decompression speed and size
    Cool! (Dont forget to check out the LZPM 0.09)

    LZTUTBO is very efficient on text files, however, check out the decompression speed at Black_Foxs test set:

    LZTURBO:
    comp speed: 729 kB/s
    dcmp speed: 3 795 kB/s

    LZPM:
    comp speed: 1 028 kB/s
    dcmp speed: 9 866 kB/s

    Different picture, isnt it? LZPM decompresses nearly 3X times faster.

    Possibly, LZTURBO was preferable tested and tuned on ENWIK8/9.

    By the way, Hamid, can you provide us some info on your compression (preferable, on -49 mode):

    Arithmetic coding, OK. Bit-oriented/byte oriented? Pure order-0/order-1 or order-1-0 PPM, or order-1-0 CM, or?

    Super-optimal parsing? OK. What this means? LZMA/CABARC-like? Does it estimate cost for each choice, or it uses constants, and so on...

    Match finder. What kind of. Hash chains? Binary tree?

    Is LZTURBO pure LZ77, or it uses some tricks, e.g. ROLZ, ... ?

    Also, its might be interesting for users and me, can you introduce yourself? For example, do you enthusiast, or maybe youre vice president of Microsoft, etc.

    Sorry for bugging you, just this forum and compression community welcomes a new member!

  26. #26
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Another quick test...

    PariahInterface.utx (-49)

    Compressed Size: 5,555,230 bytes
    Compression Time: 3094.160 Seconds
    Decompression Time: 7.053 Seconds


    PHT (-29)

    Compressed Size: 1,397,693 bytes
    Compression Time: 8.430 Seconds
    Decompression Time: 0.320 Seconds


    Scribble.wav (-49)

    Compressed Size: 17,708 bytes
    Compression Time: 109.376 Seconds
    Decompression Time: 0.348 Seconds

  27. #27
    Member
    Join Date
    Aug 2007
    Posts
    30
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hi,
    lzturbo is designed as a practical compressor for modern processors not for benchmarking.
    There is no sense to test the speed for small files, as it's not important if a user wait's 3 or 6 seconds, but there is a big difference (from the user view), between 10min or 35min. Please compare speeds only for equivalent compression ratios. Look also at the times 'real' (times users waiting for, including I/O).
    Note: This is only experimental version 0.0.1

    Now the results on my system: LZTURBO 0.0.1
    Hardware: AMD-X2-3800+ dual core. Ubuntu-Linux 7.04 32-bits:
    enwik9:
    file copy: cp enwik9 enwik9.lz real:37 user:0.12

    lzturbo using 2 cores (note:user time per core).
    lzturbo mode -11: 493689235 bytes
    comp: real:12.7 user:3.3 (per core) decomp:real 17.5 user:1.6 (per core)

    lzturbo mode -13: 378029486 bytes
    comp: real:21.3 user:16.4 decomp: real:19.7 user:2.3

    lzturbo mode -49 233322999 bytes
    comp: real:637 user:600 decomp: real:32 user:22

    lzpm 0.0.9: 242929442
    comp: real:2017 user:2002 decomp:real:66 user:53

    quicklz 1.30 final:
    mode -0: 469460173 bytes
    comp: real:22.5 user:7.7 decomp:real 20.1 user:5.9

    mode -3:411462500 bytes
    comp: real:54 user:39.2 decomp:real:18.9 user:3.8

  28. #28
    Tester

    Join Date
    May 2008
    Location
    St-Petersburg, Russia
    Posts
    182
    Thanks
    3
    Thanked 0 Times in 0 Posts
    wait for win32 version
    sorry it's here
    http://www.consultant-berater.de/lzturbo-0.0.1.tar .gz

  29. #29
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by donotdisturb
    There is no sense to test the speed for small files,
    The smallest file in my quick test was PHT.psd at 6.44 MB (6,755,163 bytes).

    Scribble.wav = 8.82 MB (9,249,480 bytes)

    PariahInterface.utx = 23.2 MB (24,375,895 bytes)


    Quote Originally Posted by donotdisturb
    Look also at the times real (times users waiting for, including I/O).
    All of my tests include I/O time.


    Quote Originally Posted by donotdisturb
    Note: This is only experimental version 0.0.1
    Looking forward to future releases!

  30. #30
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,507
    Thanks
    742
    Thanked 665 Times in 359 Posts
    Quote Originally Posted by encode
    Possibly, LZTURBO was preferable tested and tuned on ENWIK8/9.
    look at the following:

    1:byte-output (fastest decompression)
    2:bit-output
    3:not implemented
    4:best compression

    it looks like he just used my tornado model. in this case i know why its so bad on non-text files - tornado 0.1 MF doesnt find 2/3-byte strings nor contains many other improvements which improves binary files compression in rar/cab/lzma. since 0.1 ive improved binary files comporession by at least 10%

Page 1 of 2 12 LastLast

Similar Threads

  1. NEW BZP Experimental Archiver !
    By Nania Francesco in forum Data Compression
    Replies: 34
    Last Post: 20th January 2009, 21:22
  2. dcs-bwt - Experimental Burrows-Wheeler Compressor
    By Arkanosis in forum Data Compression
    Replies: 2
    Last Post: 25th June 2008, 04:53
  3. Disassembled LZTurbo.exe 0.92...
    By Raymond_NGhM in forum Forum Archive
    Replies: 10
    Last Post: 17th April 2008, 15:29
  4. LZTURBO 0.9 parallel compressor
    By donotdisturb in forum Forum Archive
    Replies: 18
    Last Post: 6th March 2008, 01:23
  5. LZTURBO 0.1 parallel compressor
    By donotdisturb in forum Forum Archive
    Replies: 5
    Last Post: 7th October 2007, 23:44

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •