Page 1 of 2 12 LastLast
Results 1 to 30 of 34

Thread: LzTurbo: The fastest now more faster

  1. #1
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    487
    Thanks
    52
    Thanked 182 Times in 133 Posts

    Exclamation LzTurbo: The fastest now more faster

    # New version :
    - the fastest compressor now more faster.
    - improved compression/decompression speed.
    - improved compression. LzTurbo beats Lzma in compression with equivalent decompression speed.
    - new TurboANX asymmetric numeral system w. JIT switching [scalar/SSE/AVX2].
    Arithmetic coding rate at byte coding speed (see entropy coder benchmark http://encode.su/threads/1890-Benchm...Entropy-Coders).
    Decoding speed [up to 1Gb/s] faster than most compressors with byte coding.
    Can decompress 10 times and more faster than lzma with a bit less compression ratio
    - Compression speed up to: 800 MB/s
    and decompression speed up to 4 GB/s per core (tested on a 2600k cpu at 4.5 GHz)
    - Standard input/output (stdin/stdout)

    Download LzTurbo v1.2 for Windows and Linux at: http://sites.google.com/site/powturbo
    Last edited by dnd; 10th August 2014 at 23:51.

  2. Thanks (3):

    avitar (8th August 2014),Jarek (8th August 2014),Sportman (8th August 2014)

  3. #2
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    487
    Thanks
    52
    Thanked 182 Times in 133 Posts
    Benchmarks

  4. #3
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    i should mention again that lzturbo from the first version violated the GPL by using the tornado 0.1 sources

  5. Thanks:

    RichSelian (12th August 2014)

  6. #4
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    I don't want to take sides on the Bulat but have reason to complain, but I think it's fair at this point that the author takes into account the possibility of open source code to remove all doubt!

  7. #5
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    789
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Input:
    916,873,928 bytes - HTML file

    Output:
    148,673,416 bytes, 16.324 sec. - 3.658 sec., tor 6 -6
    148,413,550 bytes, 10.638 sec. - 1.370 sec., lzturbo 1.2 -32 -p1
    139,969,408 bytes, 10.626 sec. - 1.548 sec., lzturbo 1.2 -32 -b1024 -p1
    136,750,016 bytes, 34.978 sec. - 3.451 sec., tor 6 -7

    153,962,336 bytes, 8.693 sec. - 1.580 sec., lzturbo 1.1 -32 -p1
    153,059,795 bytes, 13.351 sec. - 3.336 sec., tor 4 -6
    146,996,084 bytes, 19.714 sec. - 3.240 sec., tor 4 -7
    145,130,306 bytes, 9.611 sec. - 1.910 sec., lzturbo 1.1 -32 -b1024 -p1

    I don't see any similarities in results. In the business where I'm active, more then 100 individuals/companies copy your (closed source) work within 2 years. You can also feel proud...

  8. Thanks (2):

    avitar (8th August 2014),dnd (10th August 2014)

  9. #6
    Member
    Join Date
    Sep 2011
    Location
    uk
    Posts
    238
    Thanks
    188
    Thanked 17 Times in 12 Posts
    Can a 32 bit version (for win in particular, but maybe linux too) be made, to help compatibility between 32 & 64 bit PCs? TIA

    John

  10. #7
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    There is a small improvement in compression ratio and decompression speed over v1.1. However methods -30 ... -39 (all the ANS coding) give an illegal instruction error on my test machine (core i7 M620, 64 bit Ubuntu Linux). This includes both Linux compiles and also lzturbo.exe under Wine 1.6. http://mattmahoney.net/dc/text.html#1947

  11. Thanks:

    dnd (10th August 2014)

  12. #8
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    342
    Thanks
    12
    Thanked 34 Times in 28 Posts
    This does not seem to work properly

    lzturbo -32 1.sql 11.lzt
    from a 400MB-ASCII-DB-DUMP

    crash


    Code:
    Firma problema:
      Nome evento problema:    APPCRASH
      Nome applicazione:    lzturbo.exe
      Versione applicazione:    0.0.0.0
      Timestamp applicazione:    00000000
      Nome modulo con errori:    lzturbo.exe
      Versione modulo con errori:    0.0.0.0
      Timestamp modulo con errori:    00000000
      Codice eccezione:    c000001d
      Offset eccezione:    00000000000045f9
      Versione SO:    6.1.7601.2.1.0.256.1
      ID impostazioni locali:    1040
      Informazioni aggiuntive 1:    9a7b
      Ulteriori informazioni 2:    9a7b9c34b31adf51777529cc20157a02
      Ulteriori informazioni 3:    3745
      Ulteriori informazioni 4:    3745e5c2a187963f980d190eed545ddf
    Windows 7-64, 6 core, 24GB RAM (working on RAMDISK)

  13. Thanks:

    dnd (10th August 2014)

  14. #9
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    789
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Input:
    635,346,739 bytes, IIS log file

    Output:
    97,539,569 bytes, 0,808 sec. - 0.708 sec., snzip/snunzip (snappy)
    77,611,309 bytes, 0.906 sec. - 0.647 sec., qpress64 -L1T1
    73,939,113 bytes, 4.745 sec. - 0.533 sec., qpress64 -L3T1
    72,103,101 bytes, 1.371 sec. - 0.651 sec., qpress64 -L2T1
    70,512,023 bytes, 0.232 sec. - 0.284 sec., lz4 -1
    69,018,772 bytes, 0.517 sec. - 0.275 sec., lzturbo -10 -p1
    68,839,181 bytes, 0.784 sec. - 0.438 sec., lzturbo -10 -b1024 -p1
    66,947,567 bytes, 0.612 sec. - 0.732 sec., tor64 -1
    64,015,134 bytes, 0.833 sec. - 0.407 sec., lzturbo -11 -p1
    64,004,420 bytes, 1.064 sec. - 0.425 sec., lzturbo -11 -b1024 -p1
    62,327,357 bytes, 1.797 sec. - 0.406 sec., lzturbo -12 -p1
    62,316,292 bytes, 1.992 sec. - 0.425 sec., lzturbo -12 -b1024 -p1
    55,162,028 bytes, 1.169 sec. - 0.278 sec., lz4 -9
    53,358,636 bytes, 0.441 sec. - 0.304 sec., lzturbo -20 -p1
    52,065,316 bytes, 0.682 sec. - 0.621 sec., tor64 -2
    50,500,782 bytes, 0.692 sec. - 0.461 sec., lzturbo -20 -b1024 -p1
    47,332,773 bytes, 3.794 sec. - 0.264 sec., Zhuff_beta -c1 -t1
    45,788,093 bytes, 0.620 sec. - 0.420 sec., lzturbo -21 -p1
    45,509,497 bytes, 0.844 sec. - 0.446 sec., lzturbo -21 -b1024 -p1
    44,885,404 bytes, 6.522 sec. - 0.248 sec., Zhuff_beta -c2 -t1
    41,699,625 bytes, 0.671 sec. - 0.417 sec., lzturbo -30 -p1
    41,443,272 bytes, 1.928 sec. - 0.418 sec., lzturbo -22 -p1
    41,069,547 bytes, 0.969 sec. - 0.828 sec., tor64 -3
    41,033,836 bytes, 1.979 sec. - 0.439 sec., lzturbo -22 -b1024 -p1
    40,329,727 bytes, 0.617 sec. - 0.241 sec., Zhuff_beta -c0 -t1
    38,786,414 bytes, 0.847 sec. - 0.561 sec., lzturbo -30 -b1024 -p1
    38,149,852 bytes, 1.048 sec. - 0.816 sec., tor64 -4
    35,592,337 bytes, 0.777 sec. - 0.532 sec., lzturbo -31 -p1
    34,914,063 bytes, 0.981 sec. - 0.549 sec., lzturbo -31 -b1024 -p1
    33,571,803 bytes, 8.775 sec. - 1.023 sec., tor64 -11
    33,481,897 bytes, 2.361 sec. - 0.944 sec., tor64 -5
    32,125,911 bytes, 2.201 sec. - 0.520 sec., lzturbo -32 -p1
    31,732,779 bytes, 4.175 sec. - 0.925 sec., tor64 -6
    31,091,915 bytes, 2.051 sec. - 0.545 sec., lzturbo -32 -b1024 -p1
    30,366,300 bytes, 12.838 sec. - 0.994 sec., tor64 -12
    30,345,874 bytes, 14.358 sec. - 0.947 sec., tor64 -7
    29,666,149 bytes, 23.152 sec. - 0.962 sec., tor64 -13
    29,385,045 bytes, 29.062 sec. - 0.959 sec., tor64 -14
    29,341,842 bytes, 23.383 sec. - 1.032 sec., tor64 -8
    28,872,692 bytes, 38.581 sec. - 1.051 sec., tor64 -9
    28,669,942 bytes, 43.970 sec. - 1.053 sec., tor64 -10
    27,303,652 bytes, 97.900 sec. - 0.927 sec., tor64 -15
    24,346,439 bytes, 356.442 sec. - 0.889 sec., tor64 -16
    21,824,143 bytes, 715.032 sec. - 0.578 sec., lzturbo -39 -p1
    21,823,933 bytes, 714.959 sec. - 0.580 sec., lzturbo -39 -b1024 -p1
    19,219,701 bytes, 1,038.787 sec. - 1.414 sec., lzturbo -49 -p1
    19,219,701 bytes, 1,038.852 sec. - 1.404 sec., lzturbo -49 -b1024 -p1
    15,851,524 bytes, 11,233.358 sec. - 1.890 sec, Tree -t1
    Last edited by Sportman; 18th August 2014 at 19:54.

  15. Thanks (3):

    dnd (10th August 2014),Jarek (9th August 2014),Kennon Conrad (16th August 2014)

  16. #10
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    702
    Thanks
    217
    Thanked 217 Times in 134 Posts
    Great job Hamid!

    Using ANS, you can easily add some additional features:
    - checksum indicating if file was corrupted for nearly no cost: the fixed initial ANS state is the final one for decoding - it will become random if there was a damage. You can increase the probability of such damage detection as much as you need e.g. by using interleaving (e.g. even symbols coded with one state, odd with second),
    - encryption at tiny cost of speed: disturb the symbol spread in a pseudorandom way accordingly to PRNG initialized with cryptokey,
    - error correction at cost of reduced rate: add "forbidden symbol" to the alphabet and rescale the remaining ones. This symbol will not be used while encoding, but it can appear after an error while decoding - indicating it. Getting this symbol you go back and try some corrections - searching a correction tree. The larger probability of this symbol, the better correction, at cost of worse rate. I have just implemented analogous approach for deletion channel (we don't even know the capacity for it) and it turns out having much better performance than e.g. LDPC-based methods: https://github.com/JarekDuda/DeletionChannelPracticalCorrection

    Ps. How to improve symbol spread: http://encode.su/threads/2013-Asymme...-symbol-spread

    Ps2. JamesB suggests that order 1 rANS would be only twice slower than FSE/tANS: http://encode.su/threads/1920-In-mem...ll=1#post37574
    How would order 1 entropy coding improve compression rate for LZ?
    Last edited by Jarek; 12th August 2014 at 09:13.

  17. Thanks:

    dnd (10th August 2014)

  18. #11
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    LZTurbo v.1.2 is in testing with WCC 2014 Benchmark.
    With windows 7 - intel Core i7 920.
    with options -30 -31 -32 -40 -41 -42 detected errors or program go in crush !
    @ sportman
    Input: 635,346,739 bytes, IIS log file
    LZA v.010 ?

  19. Thanks:

    dnd (10th August 2014)

  20. #12
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    789
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Quote Originally Posted by Nania Francesco View Post
    Input: 635,346,739 bytes, IIS log file
    LZA v.010 ?
    38,103,866 bytes, 20.596 sec. - 1.455 sec., lza -m1 -t1
    36,146,007 bytes, 22.552 sec. - 1.445 sec., lza -m2 -t1
    35,047,162 bytes, 28.212 sec. - 1.447 sec., lza -m3 -t1
    35,555,904 bytes, 29.211 sec. - 1.459 sec., lza -m4 -t1
    36,585,762 bytes, 31.030 sec. - 1.466 sec., lza -m5 -t1

    37,295,306 bytes, 22.433 sec. - 1.447 sec., lza -mx1 -t1
    35,455,522 bytes, 24.514 sec. - 1.437 sec., lza -mx2 -t1
    34,749,729 bytes, 30.142 sec. - 1.447 sec., lza -mx3 -t1
    35,350,029 bytes, 31.241 sec. - 1.460 sec., lza -mx4 -t1
    36,388,619 bytes, 33.163 sec. - 1.464 sec., lza -mx5 -t1

  21. Thanks:

    Nania Francesco (10th August 2014)

  22. #13
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    487
    Thanks
    52
    Thanked 182 Times in 133 Posts
    I've uploaded a new build with better cpu detection.


    @Sportman:
    thank you very much for your time and efforts testing lzturbo. Please include, if possible option "-39 -b1024"


    @Jarek:
    Thanks for your words of support and suggestions. Thank you also for your outstanding invention ANS.


    @Matt
    Also thanks for your valuable time testing lzturbo. Please, if possible test again the options -30,-31,-32,-39 again.


    @fcorbelli
    Thanks considering lzturbo in your DB compression. Please use the new build for testing again.


    @Nania
    Thank for considering lzturbo in MOC. Please retest with the new build.
    The options (-40, 41, -42) are not available and a message error is displayed.
    Please use the default blocksize or a blocksize >64 (ex. -b128 or -b1024) and do not specify -b29 (blocksize=29MB).
    I must remark that your timings are very volatile, maybe you can consider testing with a ramdisk or SSD in the future.

    @avita:
    i'm considering making a 32 bits in the near future.

  23. Thanks:

    avitar (11th August 2014)

  24. #14
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    789
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Quote Originally Posted by dnd View Post
    Please include, if possible option "-39 -b1024"
    Added.

    If it's possible, can lzturbo get an option where output file name can be specified.

  25. #15
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    487
    Thanks
    52
    Thanked 182 Times in 133 Posts
    Thanks for your quick response. You can now also specify a file or a directory as destination. See readme file.

  26. #16
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    The Aug. 10 build still gives me illegal instruction errors for -30, -31, -32, -39.

  27. #17
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    487
    Thanks
    52
    Thanked 182 Times in 133 Posts
    Sorry Matt, there is a new build now. If the problem persists, you can use the option -S (disable SIMD for ANS).

  28. #18
    Member
    Join Date
    Jun 2013
    Location
    Sweden
    Posts
    150
    Thanks
    9
    Thanked 25 Times in 23 Posts
    lzturbo 1.2 Copyright (c) 2007-2014 Hamid Buzidi Aug 10 2014
    Laptop i7-3610qm (2300mhz/turbo 3100mhz) 16GB ram (1600mhz) (ramdisk takes 3GB)
    compressing DVD-movie of 4.5GB to/from 1.5TB with USB 2.0
    iso = 4.699.740.160
    rar5 = ~11.5 minutes using ~6GB memory = 4.425.651.858 bytes
    lzt -49 -p1 = ~58 minutes using ~4GB memory (stupid me deleted it before I fell asleep yesterday but I think it´s the same as p2)
    lzt -49 -p2 = ~33 minutes using ~7GB memory = 4.375.808.992 bytes
    lzt -49 -p3 = ~24 minutes using ~10GB memory = 4.375.808.992 (identical to p2)
    lzt -49 p8 = ~11 minutes using ~3GB memory = 4.421.355.653 bytes
    lzt -39 p1 = ~52 minutes using ~4GB memory = 4.426.575.839 bytes
    lzt -39 p3 = ~21 minutes using ~10GB memory = 4.426.574.781 bytes - equal to p1 until 347.591.936 bytes, that is about first block written to disk
    7z (ultra) LZMA2 1024MB fb=273 mc=999 threads=3 = 4.424.035.537 bytes @ ~44 minutes using 10GB memory
    -49-p1 decompressed ok (crc32 matched) in ~4 minutes
    -39-p3 decompressed ok in ~2 minutes (total .lzt file was in windows cache) and compared flawless with original.
    Program does not save time/date of file. I suggest to author to set default output to be .\ in case user omits it.
    Last edited by a902cd23; 11th August 2014 at 14:13. Reason: correction

  29. Thanks:

    dnd (11th August 2014)

  30. #19
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    It works now. No need to use -S.

  31. Thanks:

    dnd (11th August 2014)

  32. #20
    Member
    Join Date
    Sep 2011
    Location
    uk
    Posts
    238
    Thanks
    188
    Thanked 17 Times in 12 Posts
    Been trying latest version, quad amd 4300, 4.3Ghz, win 7. Compressing a 4Gbyte .img (an image of a linux jessie raspberry pi system). Few comments...

    1) Tried -22 & compresses to 1.8Gbytes in about 60 secs. arc -mx gets it to 1.3Gbytes but it takes 1000+ secs. Not sure what'd be better settings for better compression. Uses all 4 processors, about 60% of each processor's time.

    2) no console output- would like a progress indicator (as option) as % or at least a . every n secs, and ideally an eta indication for current file

    3) and a summary line saying eg took:xx secs, file:xxx.lzt size 1234567 Mbytes

    4) -49 used 100% of the 4 processors, but almost stopped the system, eg keyboard working properly, so I aborted it. What'd be better settings?

    5) think someone else requested an explicit output filename instead of just a directory. This'd be good.

    6) think you should increment the version no eg 1.21, 1.22 etc for _each_ change you make so we/you can be 100% sure which version we have- think the date is a bit confusing.

    7) maybe just a few lines of extra help/guidance about what the impact of changing the
    the -m & -b values could be added to the lzturbo help eg re method, level & blocksize

    So overall looks good, thanks.

    John
    Last edited by avitar; 12th August 2014 at 18:47. Reason: more qs

  33. #21
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    487
    Thanks
    52
    Thanked 182 Times in 133 Posts
    Quote Originally Posted by Jarek View Post
    How would order 1 entropy coding improve compression rate for LZ?
    for lz with bytewise coders like ANS there is no much to expect in compression improvement specially for binary files. Normally you must add file type detection then specify the literal context (part of previous 1 or 2 bytes).
    Using "order 1" for all files can hurt compression on binary files. The speed can also be affected, specially for fast decompression.

  34. #22
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    487
    Thanks
    52
    Thanked 182 Times in 133 Posts
    Quote Originally Posted by avitar View Post
    Been trying latest version, quad amd 4300, 4.3Ghz, win 7. Compressing a 4Gbyte .img (an image of a linux jessie raspberry pi system). Few comments...
    1) Normally there is nothing special to set when using modes (-10,-11,-12,...-31,-32).
    You can specify the block size, if you want (ex. -b128 or -b1024) for large files and better compression.
    For best compression in the fast modes use "-32" (ex. "lzturbo -32 c:\tmp\* bak_dir" ).


    2) and 3) Yes, this will change in the next version.


    4) You can specify -p0 to disable multithreading (ex. lzturbo -39 -b256 -p0 file file.lzt" for your system w. 4GB memory).
    Level 9 (-19,-29,-39,-49) uses optimal parsing like 7-zip and is very slow, but with the best compression.
    Level 9 needs ~13N (N = block size ) memory per thread.


    5) think someone else requested an explicit output filename instead of just a directory. This'd be good.
    You can also specify a destination file in the current version (ex. "lzturbo -32 c:\tmp\input input.lzt" ).


    6) for me it is better, to use the same version for small fixes.


    7) - For best compression in the fast modes use "-32".
    - If you use you level 9 (-19,-29,-39,-49) with large files, better specify -p0.

  35. Thanks:

    avitar (13th August 2014)

  36. #23
    Member
    Join Date
    Sep 2011
    Location
    uk
    Posts
    238
    Thanks
    188
    Thanked 17 Times in 12 Posts
    Thanks.

    Re 5 don't see that explained in the help. Suggest that your other answers, editted, could be added to the help or to a 2nd page of help, with the examples.

    Also re my 3, add ' max memory used' so impact of -b can be seen.

    John

  37. #24
    Member
    Join Date
    May 2008
    Location
    brazil
    Posts
    163
    Thanks
    0
    Thanked 3 Times in 3 Posts
    I don t know the real value of lzturbo. Dnd does not release any paper or some code using open source license.

    I prefer freearc.Opensource is more hard to to get money ,but cost much less and the code can be reused without to pay licenses.

  38. #25
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    467
    Thanks
    149
    Thanked 160 Times in 108 Posts
    Quote Originally Posted by Jarek View Post
    Ps2. JamesB suggests that order 1 rANS would be only twice slower than FSE/tANS: http://encode.su/threads/1920-In-mem...ll=1#post37574
    How would order 1 entropy coding improve compression rate for LZ?
    I've been away for a couple weeks, but a couple examples using LZ4 and Snappy (via snzip), both of which I think produce byte-aligned output streams with no entropy encoding of their own. This is a trivial example, but maybe gives an indication.

    jkb@seq3a[tmp/snzip-1....] ./snzip < ~/scratch/data/enwik9 > ../en9.snappy
    jkb@seq3a[tmp/snzip-1....] ls -l ../en9.snappy
    -rw-r--r-- 1 jkb team117 527939908 Aug 18 14:31 ../en9.snappy
    jkb@seq3a[tmp/snzip-1....] ~/work/compression/rANS_static4 -o0 < /tmp/en9.snappy | wc -c
    Took 2698054 microseconds, 195.7 MB/s
    462867915
    jkb@seq3a[tmp/snzip-1....] ~/work/compression/rANS_static4 -o1 < /tmp/en9.snappy | wc -c
    Took 6108508 microseconds, 86.4 MB/s
    443909486

    jkb@seq3a[compress.../lz4-read...] ./lz4demo32.exe ~/scratch/data/enwik9 /tmp/en9.lz4
    *** Compression CLI using LZ4 algorithm , by Yann Collet (Jul 26 2012) ***
    Compressed 1000000000 bytes into 507213217 bytes ==> 50.72%
    Done in 4.08 s ==> 233.74 MB/s
    jkb@seq3a[compress.../lz4-read...] ls -l /tmp/en9.lz4
    -rw-r--r-- 1 jkb team117 507213217 Aug 18 14:32 /tmp/en9.lz4
    jkb@seq3a[compress.../lz4-read...] ~/work/compression/rANS_static4 -o0 < /tmp/en9.lz4 | wc -c
    Took 2711335 microseconds, 187.1 MB/s
    435639681
    jkb@seq3a[compress.../lz4-read...] ~/work/compression/rANS_static4 -o1 < /tmp/en9.lz4 | wc -c
    Took 6112700 microseconds, 83.0 MB/s
    424334289

    (The lz4 is an old copy, so I've no idea how it compares speed wise since I downloaded it.)

    Anyway the short answer is around 3-4% saving only going from O(0) to O(1) entropy encoding. I expect there are FAR better ways to do this though.

    Some timings:
    lz4 alone: 4.31sec enc, 0.93sec dec.
    lz4 + rans_O(0): 4.45sec enc, 2.76sec dec
    lz4 + rans_O(1): 6.38sec enc, 6.94sec dec

    So it's still a major hit for decoding performance. Timings here are lz4 | mbuffer | rans or rans | mbuffer | lz4. The mbuffer permits it to use a 2Mb (default) memory buffer to streamline the unix pipes and improve parallel performance. This order-0 decoder is only going at ~240 MB/sec though so FSE or one of Ryg's 64-bit variants would run faster. The only reason I stuck with this version for my work is I wanted something that was identical between 32-bit and 64-bit platforms and it was Good Enough(TM) for the intended use.

    James

    PS. The rANS_static4 source is ftp://ftp.sanger.ac.uk/pub/users/jkb/rANS_static4.c and based on Ryg's implementation with more aggressive unrolling.

  39. Thanks:

    Jarek (18th August 2014)

  40. #26
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    789
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Input:
    691,759,309 bytes, unsorted list with almost 11mln URLs (30% unique).

    Output:
    253,831,058 bytes, 1.762 sec. - 1.472 sec., tor64 -1
    244,090,127 bytes, 1.471 sec. - 0.975 sec., snzip/snunzip (snappy)
    229,231,586 bytes, 1.756 sec. - 1.359 sec., qpress64 -L1T1
    228,731,049 bytes, 0.344 sec. - 0.340 sec., lz4 -1
    225,591,975 bytes, 1.703 sec. - 0.413 sec., lzturbo -10 -p1
    213,704,697 bytes, 2.891 sec. - 1.389 sec., qpress64 -L2T1
    209,667,579 bytes, 2.638 sec. - 0.553 sec., lzturbo -11 -p1
    208,379,530 bytes, 2.166 sec. - 1.341 sec., tor64 -2
    204,277,427 bytes, 7.560 sec. - 1.068 sec., qpress64 -L3T1
    202,709,615 bytes, 6.581 sec. - 0.553 sec., lzturbo -12 -p1
    199,042,115 bytes, 1.594 sec. - 0.612 sec., lzturbo -20 -p1
    187,260,122 bytes, 2.097 sec. - 0.330 sec., lz4 -9
    185,809,810 bytes, 515.495 sec. - 0.610 sec., lzturbo -19 -p1
    166,478,661 bytes, 3.432 sec. - 2.308 sec., tor64 -3
    159,816,868 bytes, 2.293 sec. - 0.976 sec., lzturbo -30 -p1
    153,453,669 bytes, 9.634 sec. - 4.047 sec., zhuff_beta -c1 -t1
    148,006,274 bytes, 14.177 sec. - 0.628 sec., zhuff_beta -c2 -t1
    145,879,704 bytes, 2.066 sec. - 0.610 sec., zhuff_beta -c0 -t1
    139,594,362 bytes, 1.970 sec. - 0.665 sec., lzturbo -21 -p1
    132,484,415 bytes, 3.446 sec. - 1.938 sec., tor64 -4
    113,994,789 bytes, 5.757 sec. - 0.631 sec., lzturbo -22 -p1
    111,101,281 bytes, 2.353 sec. - 0.930 sec., lzturbo -31 -p1
    100,787,832 bytes, 6.946 sec. - 2.280 sec., tor64 -5
    84,223,767 bytes, 6.036 sec. - 0.849 sec., lzturbo -32 -p1
    80,364,741 bytes, 9.534 sec. - 1.928 sec., tor64 -6
    76,132,986 bytes, 23.006 sec. - 1.901 sec., tor64 -11
    70,167,371 bytes, 34.920 sec. - 1.800 sec., tor64 -12
    69,537,310 bytes, 20.230 sec. - 1.763 sec., tor64 -7
    67,239,044 bytes, 56.043 sec. - 1.747 sec., tor64 -13
    65,988,172 bytes, 69.410 sec. - 1.754 sec., tor64 -14
    64,155,952 bytes, 119.300 sec. - 1.721 sec., tor64 -15
    63,841,180 bytes, 32.758 sec. - 1.767 sec., tor64 -8
    62,863,233 bytes, 394.516 sec. - 0.681 sec., lzturbo -29 -p1
    62,267,140 bytes, 51.723 sec. - 1.764 sec., tor64 -9
    61,110,352 bytes, 61.808 sec. - 1.769 sec., tor64 -10
    60,187,593 bytes, 260.807 sec. - 1.665 sec., tor64 -16
    51,475,718 bytes, 456.172 sec. - 0.844 sec., lzturbo -39 -p1
    48,497,129 bytes, 654.649 sec. - 2.716 sec., lzturbo -49 -p1

  41. Thanks:

    dnd (21st August 2014)

  42. #27
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    342
    Thanks
    12
    Thanked 34 Times in 28 Posts
    stdin-stdout does not seems to work fine on Windows.

    something like that
    Code:
    mysqldump -uroot -ppassword database | gzip >1.gz
    is OK
    but
    Code:
    mysqldump -uroot -ppassword database | lzturbo -of -10 >1.lzt
    throw an error
    Code:
    mysqldump: Got errno 22 on write

  43. Thanks:

    dnd (24th August 2014)

  44. #28
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    789
    Thanks
    64
    Thanked 274 Times in 192 Posts
    Quote Originally Posted by dnd View Post
    @avita:
    i'm considering making a 32 bits in the near future.
    I use LzTurbo to compress one kind of data where it's better then any other kind of compressor in speed enc/dec/size combination. Problem I face is that sometimes I need to run it on 32-bits Win OS and need to switch back to an other compressor I used before, what is much less optimal and use double time enc/dec and create a bigger output. Any change to compile a 32-bits version of LzTurbo, I use only -32 -b1024 -p1 to compress and -d -p1 to decompress?

  45. #29
    Member
    Join Date
    Jun 2013
    Location
    Sweden
    Posts
    150
    Thanks
    9
    Thanked 25 Times in 23 Posts
    It looks like LzTurbo cannot handle files opened by other files, if I open a DVD-movie (ISO-file) with VLC and then try to compress it I only get a 32bytes file. After closing VLC LzTurbo can handle file.

  46. #30
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    487
    Thanks
    52
    Thanked 182 Times in 133 Posts
    @Sportman: lookt at PM

    @a902cd23: thanks for reporting. I'll look at it in the next LzTurbo update

Page 1 of 2 12 LastLast

Similar Threads

  1. Faster permutation inverse algorithm?
    By nburns in forum The Off-Topic Lounge
    Replies: 7
    Last Post: 21st July 2013, 12:28
  2. Computing the BWTS faster
    By nburns in forum Data Compression
    Replies: 4
    Last Post: 20th July 2013, 21:17
  3. Replies: 1
    Last Post: 21st June 2011, 12:48
  4. GCC 4.6 faster than previous GCC, faster than LLVM
    By willvarfar in forum The Off-Topic Lounge
    Replies: 2
    Last Post: 15th November 2010, 16:09
  5. Faster PAQ7/8
    By LovePimple in forum Forum Archive
    Replies: 6
    Last Post: 8th February 2007, 14:04

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •