LzTurbo: The fastest now more faster

[dnd]

# New version :
- the fastest compressor now more faster.
- improved compression/decompression speed.
- improved compression. LzTurbo beats Lzma in compression with equivalent decompression speed.
- new TurboANX asymmetric numeral system w. JIT switching [scalar/SSE/AVX2].
Arithmetic coding rate at byte coding speed (see entropy coder benchmark http://encode.su/threads/1890-Benchm...Entropy-Coders).
Decoding speed [up to 1Gb/s] faster than most compressors with byte coding.
Can decompress 10 times and more faster than lzma with a bit less compression ratio
- Compression speed up to: 800 MB/s
and decompression speed up to 4 GB/s per core (tested on a 2600k cpu at 4.5 GHz)
- Standard input/output (stdin/stdout)

Download LzTurbo v1.2 for Windows and Linux at: http://sites.google.com/site/powturbo

[dnd]

Benchmarks

[Bulat Ziganshin]

i should mention again that lzturbo from the first version violated the GPL by using the tornado 0.1 sources

[Nania Francesco]

I don't want to take sides on the Bulat but have reason to complain, but I think it's fair at this point that the author takes into account the possibility of open source code to remove all doubt!

[Sportman]

Input:
916,873,928 bytes - HTML file

Output:
148,673,416 bytes, 16.324 sec. - 3.658 sec., tor 6 -6
148,413,550 bytes, 10.638 sec. - 1.370 sec., lzturbo 1.2 -32 -p1
139,969,408 bytes, 10.626 sec. - 1.548 sec., lzturbo 1.2 -32 -b1024 -p1
136,750,016 bytes, 34.978 sec. - 3.451 sec., tor 6 -7

153,962,336 bytes, 8.693 sec. - 1.580 sec., lzturbo 1.1 -32 -p1
153,059,795 bytes, 13.351 sec. - 3.336 sec., tor 4 -6
146,996,084 bytes, 19.714 sec. - 3.240 sec., tor 4 -7
145,130,306 bytes, 9.611 sec. - 1.910 sec., lzturbo 1.1 -32 -b1024 -p1

I don't see any similarities in results. In the business where I'm active, more then 100 individuals/companies copy your (closed source) work within 2 years. You can also feel proud...

[avitar]

Can a 32 bit version (for win in particular, but maybe linux too) be made, to help compatibility between 32 & 64 bit PCs? TIA

John

[Matt Mahoney]

There is a small improvement in compression ratio and decompression speed over v1.1. However methods -30 ... -39 (all the ANS coding) give an illegal instruction error on my test machine (core i7 M620, 64 bit Ubuntu Linux). This includes both Linux compiles and also lzturbo.exe under Wine 1.6. http://mattmahoney.net/dc/text.html#1947

[fcorbelli]

This does not seem to work properly

lzturbo -32 1.sql 11.lzt
from a 400MB-ASCII-DB-DUMP

crash

Firma problema:
  Nome evento problema:    APPCRASH
  Nome applicazione:    lzturbo.exe
  Versione applicazione:    0.0.0.0
  Timestamp applicazione:    00000000
  Nome modulo con errori:    lzturbo.exe
  Versione modulo con errori:    0.0.0.0
  Timestamp modulo con errori:    00000000
  Codice eccezione:    c000001d
  Offset eccezione:    00000000000045f9
  Versione SO:    6.1.7601.2.1.0.256.1
  ID impostazioni locali:    1040
  Informazioni aggiuntive 1:    9a7b
  Ulteriori informazioni 2:    9a7b9c34b31adf51777529cc20157a02
  Ulteriori informazioni 3:    3745
  Ulteriori informazioni 4:    3745e5c2a187963f980d190eed545ddf

Windows 7-64, 6 core, 24GB RAM (working on RAMDISK)

[Sportman]

Input:
635,346,739 bytes, IIS log file

Output:
97,539,569 bytes, 0,808 sec. - 0.708 sec., snzip/snunzip (snappy)
77,611,309 bytes, 0.906 sec. - 0.647 sec., qpress64 -L1T1
73,939,113 bytes, 4.745 sec. - 0.533 sec., qpress64 -L3T1
72,103,101 bytes, 1.371 sec. - 0.651 sec., qpress64 -L2T1
70,512,023 bytes, 0.232 sec. - 0.284 sec., lz4 -1
69,018,772 bytes, 0.517 sec. - 0.275 sec., lzturbo -10 -p1
68,839,181 bytes, 0.784 sec. - 0.438 sec., lzturbo -10 -b1024 -p1
66,947,567 bytes, 0.612 sec. - 0.732 sec., tor64 -1
64,015,134 bytes, 0.833 sec. - 0.407 sec., lzturbo -11 -p1
64,004,420 bytes, 1.064 sec. - 0.425 sec., lzturbo -11 -b1024 -p1
62,327,357 bytes, 1.797 sec. - 0.406 sec., lzturbo -12 -p1
62,316,292 bytes, 1.992 sec. - 0.425 sec., lzturbo -12 -b1024 -p1
55,162,028 bytes, 1.169 sec. - 0.278 sec., lz4 -9
53,358,636 bytes, 0.441 sec. - 0.304 sec., lzturbo -20 -p1
52,065,316 bytes, 0.682 sec. - 0.621 sec., tor64 -2
50,500,782 bytes, 0.692 sec. - 0.461 sec., lzturbo -20 -b1024 -p1
47,332,773 bytes, 3.794 sec. - 0.264 sec., Zhuff_beta -c1 -t1
45,788,093 bytes, 0.620 sec. - 0.420 sec., lzturbo -21 -p1
45,509,497 bytes, 0.844 sec. - 0.446 sec., lzturbo -21 -b1024 -p1
44,885,404 bytes, 6.522 sec. - 0.248 sec., Zhuff_beta -c2 -t1
41,699,625 bytes, 0.671 sec. - 0.417 sec., lzturbo -30 -p1
41,443,272 bytes, 1.928 sec. - 0.418 sec., lzturbo -22 -p1
41,069,547 bytes, 0.969 sec. - 0.828 sec., tor64 -3
41,033,836 bytes, 1.979 sec. - 0.439 sec., lzturbo -22 -b1024 -p1
40,329,727 bytes, 0.617 sec. - 0.241 sec., Zhuff_beta -c0 -t1
38,786,414 bytes, 0.847 sec. - 0.561 sec., lzturbo -30 -b1024 -p1
38,149,852 bytes, 1.048 sec. - 0.816 sec., tor64 -4
35,592,337 bytes, 0.777 sec. - 0.532 sec., lzturbo -31 -p1
34,914,063 bytes, 0.981 sec. - 0.549 sec., lzturbo -31 -b1024 -p1
33,571,803 bytes, 8.775 sec. - 1.023 sec., tor64 -11
33,481,897 bytes, 2.361 sec. - 0.944 sec., tor64 -5
32,125,911 bytes, 2.201 sec. - 0.520 sec., lzturbo -32 -p1
31,732,779 bytes, 4.175 sec. - 0.925 sec., tor64 -6
31,091,915 bytes, 2.051 sec. - 0.545 sec., lzturbo -32 -b1024 -p1
30,366,300 bytes, 12.838 sec. - 0.994 sec., tor64 -12
30,345,874 bytes, 14.358 sec. - 0.947 sec., tor64 -7
29,666,149 bytes, 23.152 sec. - 0.962 sec., tor64 -13
29,385,045 bytes, 29.062 sec. - 0.959 sec., tor64 -14
29,341,842 bytes, 23.383 sec. - 1.032 sec., tor64 -8
28,872,692 bytes, 38.581 sec. - 1.051 sec., tor64 -9
28,669,942 bytes, 43.970 sec. - 1.053 sec., tor64 -10
27,303,652 bytes, 97.900 sec. - 0.927 sec., tor64 -15
24,346,439 bytes, 356.442 sec. - 0.889 sec., tor64 -16
21,824,143 bytes, 715.032 sec. - 0.578 sec., lzturbo -39 -p1
21,823,933 bytes, 714.959 sec. - 0.580 sec., lzturbo -39 -b1024 -p1
19,219,701 bytes, 1,038.787 sec. - 1.414 sec., lzturbo -49 -p1
19,219,701 bytes, 1,038.852 sec. - 1.404 sec., lzturbo -49 -b1024 -p1
15,851,524 bytes, 11,233.358 sec. - 1.890 sec, Tree -t1

[Jarek]

Great job Hamid!

Using ANS, you can easily add some additional features:
- checksum indicating if file was corrupted for nearly no cost: the fixed initial ANS state is the final one for decoding - it will become random if there was a damage. You can increase the probability of such damage detection as much as you need e.g. by using interleaving (e.g. even symbols coded with one state, odd with second),
- encryption at tiny cost of speed: disturb the symbol spread in a pseudorandom way accordingly to PRNG initialized with cryptokey,
- error correction at cost of reduced rate: add "forbidden symbol" to the alphabet and rescale the remaining ones. This symbol will not be used while encoding, but it can appear after an error while decoding - indicating it. Getting this symbol you go back and try some corrections - searching a correction tree. The larger probability of this symbol, the better correction, at cost of worse rate. I have just implemented analogous approach for deletion channel (we don't even know the capacity for it) and it turns out having much better performance than e.g. LDPC-based methods: https://github.com/JarekDuda/DeletionChannelPracticalCorrection

Ps. How to improve symbol spread: http://encode.su/threads/2013-Asymme...-symbol-spread

Ps2. JamesB suggests that order 1 rANS would be only twice slower than FSE/tANS: http://encode.su/threads/1920-In-mem...ll=1#post37574
How would order 1 entropy coding improve compression rate for LZ?

[Nania Francesco]

LZTurbo v.1.2 is in testing with WCC 2014 Benchmark.
With windows 7 - intel Core i7 920.
with options -30 -31 -32 -40 -41 -42 detected errors or program go in crush !
@ sportman
Input: 635,346,739 bytes, IIS log file
LZA v.010 ?

[Sportman]

Input: 635,346,739 bytes, IIS log file
LZA v.010 ?

38,103,866 bytes, 20.596 sec. - 1.455 sec., lza -m1 -t1
36,146,007 bytes, 22.552 sec. - 1.445 sec., lza -m2 -t1
35,047,162 bytes, 28.212 sec. - 1.447 sec., lza -m3 -t1
35,555,904 bytes, 29.211 sec. - 1.459 sec., lza -m4 -t1
36,585,762 bytes, 31.030 sec. - 1.466 sec., lza -m5 -t1

37,295,306 bytes, 22.433 sec. - 1.447 sec., lza -mx1 -t1
35,455,522 bytes, 24.514 sec. - 1.437 sec., lza -mx2 -t1
34,749,729 bytes, 30.142 sec. - 1.447 sec., lza -mx3 -t1
35,350,029 bytes, 31.241 sec. - 1.460 sec., lza -mx4 -t1
36,388,619 bytes, 33.163 sec. - 1.464 sec., lza -mx5 -t1

[dnd]

I've uploaded a new build with better cpu detection.


@Sportman:
thank you very much for your time and efforts testing lzturbo. Please include, if possible option "-39 -b1024"


@Jarek:
Thanks for your words of support and suggestions. Thank you also for your outstanding invention ANS.


@Matt
Also thanks for your valuable time testing lzturbo. Please, if possible test again the options -30,-31,-32,-39 again.


@fcorbelli
Thanks considering lzturbo in your DB compression. Please use the new build for testing again.


@Nania
Thank for considering lzturbo in MOC. Please retest with the new build.
The options (-40, 41, -42) are not available and a message error is displayed.
Please use the default blocksize or a blocksize >64 (ex. -b128 or -b1024) and do not specify -b29 (blocksize=29MB).
I must remark that your timings are very volatile, maybe you can consider testing with a ramdisk or SSD in the future.

@avita:
i'm considering making a 32 bits in the near future.

[Sportman]

Please include, if possible option "-39 -b1024"

Added.

If it's possible, can lzturbo get an option where output file name can be specified.

[dnd]

Thanks for your quick response. You can now also specify a file or a directory as destination. See readme file.

[Matt Mahoney]

The Aug. 10 build still gives me illegal instruction errors for -30, -31, -32, -39.

[dnd]

Sorry Matt, there is a new build now. If the problem persists, you can use the option -S (disable SIMD for ANS).

[a902cd23]

lzturbo 1.2 Copyright (c) 2007-2014 Hamid Buzidi Aug 10 2014
Laptop i7-3610qm (2300mhz/turbo 3100mhz) 16GB ram (1600mhz) (ramdisk takes 3GB)
compressing DVD-movie of 4.5GB to/from 1.5TB with USB 2.0
iso = 4.699.740.160
rar5 = ~11.5 minutes using ~6GB memory = 4.425.651.858 bytes
lzt -49 -p1 = ~58 minutes using ~4GB memory (stupid me deleted it before I fell asleep yesterday but I think it´s the same as p2)
lzt -49 -p2 = ~33 minutes using ~7GB memory = 4.375.808.992 bytes
lzt -49 -p3 = ~24 minutes using ~10GB memory = 4.375.808.992 (identical to p2)
lzt -49 p8 = ~11 minutes using ~3GB memory = 4.421.355.653 bytes
lzt -39 p1 = ~52 minutes using ~4GB memory = 4.426.575.839 bytes
lzt -39 p3 = ~21 minutes using ~10GB memory = 4.426.574.781 bytes - equal to p1 until 347.591.936 bytes, that is about first block written to disk
7z (ultra) LZMA2 1024MB fb=273 mc=999 threads=3 = 4.424.035.537 bytes @ ~44 minutes using 10GB memory
-49-p1 decompressed ok (crc32 matched) in ~4 minutes
-39-p3 decompressed ok in ~2 minutes (total .lzt file was in windows cache) and compared flawless with original.
Program does not save time/date of file. I suggest to author to set default output to be .\ in case user omits it.

[Matt Mahoney]

It works now. No need to use -S.

[avitar]

Been trying latest version, quad amd 4300, 4.3Ghz, win 7. Compressing a 4Gbyte .img (an image of a linux jessie raspberry pi system). Few comments...

1) Tried -22 & compresses to 1.8Gbytes in about 60 secs. arc -mx gets it to 1.3Gbytes but it takes 1000+ secs. Not sure what'd be better settings for better compression. Uses all 4 processors, about 60% of each processor's time.

2) no console output- would like a progress indicator (as option) as % or at least a . every n secs, and ideally an eta indication for current file

3) and a summary line saying eg took:xx secs, file:xxx.lzt size 1234567 Mbytes

4) -49 used 100% of the 4 processors, but almost stopped the system, eg keyboard working properly, so I aborted it. What'd be better settings?

5) think someone else requested an explicit output filename instead of just a directory. This'd be good.

6) think you should increment the version no eg 1.21, 1.22 etc for _each_ change you make so we/you can be 100% sure which version we have- think the date is a bit confusing.

7) maybe just a few lines of extra help/guidance about what the impact of changing the
the -m & -b values could be added to the lzturbo help eg re method, level & blocksize

So overall looks good, thanks.

John

[dnd]

How would order 1 entropy coding improve compression rate for LZ?

for lz with bytewise coders like ANS there is no much to expect in compression improvement specially for binary files. Normally you must add file type detection then specify the literal context (part of previous 1 or 2 bytes).
Using "order 1" for all files can hurt compression on binary files. The speed can also be affected, specially for fast decompression.

[dnd]

Been trying latest version, quad amd 4300, 4.3Ghz, win 7. Compressing a 4Gbyte .img (an image of a linux jessie raspberry pi system). Few comments...

1) Normally there is nothing special to set when using modes (-10,-11,-12,...-31,-32).
You can specify the block size, if you want (ex. -b128 or -b1024) for large files and better compression.
For best compression in the fast modes use "-32" (ex. "lzturbo -32 c:\tmp\* bak_dir" ).


2) and 3) Yes, this will change in the next version.


4) You can specify -p0 to disable multithreading (ex. lzturbo -39 -b256 -p0 file file.lzt" for your system w. 4GB memory).
Level 9 (-19,-29,-39,-49) uses optimal parsing like 7-zip and is very slow, but with the best compression.
Level 9 needs ~13N (N = block size ) memory per thread.


5) think someone else requested an explicit output filename instead of just a directory. This'd be good.
You can also specify a destination file in the current version (ex. "lzturbo -32 c:\tmp\input input.lzt" ).


6) for me it is better, to use the same version for small fixes.


7) - For best compression in the fast modes use "-32".
- If you use you level 9 (-19,-29,-39,-49) with large files, better specify -p0.

[avitar]

Thanks.

Re 5 don't see that explained in the help. Suggest that your other answers, editted, could be added to the help or to a 2nd page of help, with the examples.

Also re my 3, add ' max memory used' so impact of -b can be seen.

John

[lunaris]

I don t know the real value of lzturbo. Dnd does not release any paper or some code using open source license.

I prefer freearc.Opensource is more hard to to get money ,but cost much less and the code can be reused without to pay licenses.

[JamesB]

Ps2. JamesB suggests that order 1 rANS would be only twice slower than FSE/tANS: http://encode.su/threads/1920-In-mem...ll=1#post37574
How would order 1 entropy coding improve compression rate for LZ?

I've been away for a couple weeks, but a couple examples using LZ4 and Snappy (via snzip), both of which I think produce byte-aligned output streams with no entropy encoding of their own. This is a trivial example, but maybe gives an indication.

jkb@seq3a[tmp/snzip-1....] ./snzip < ~/scratch/data/enwik9 > ../en9.snappy
jkb@seq3a[tmp/snzip-1....] ls -l ../en9.snappy
-rw-r--r-- 1 jkb team117 527939908 Aug 18 14:31 ../en9.snappy
jkb@seq3a[tmp/snzip-1....] ~/work/compression/rANS_static4 -o0 < /tmp/en9.snappy | wc -c
Took 2698054 microseconds, 195.7 MB/s
462867915
jkb@seq3a[tmp/snzip-1....] ~/work/compression/rANS_static4 -o1 < /tmp/en9.snappy | wc -c
Took 6108508 microseconds, 86.4 MB/s
443909486

jkb@seq3a[compress.../lz4-read...] ./lz4demo32.exe ~/scratch/data/enwik9 /tmp/en9.lz4
*** Compression CLI using LZ4 algorithm , by Yann Collet (Jul 26 2012) ***
Compressed 1000000000 bytes into 507213217 bytes ==> 50.72%
Done in 4.08 s ==> 233.74 MB/s
jkb@seq3a[compress.../lz4-read...] ls -l /tmp/en9.lz4
-rw-r--r-- 1 jkb team117 507213217 Aug 18 14:32 /tmp/en9.lz4
jkb@seq3a[compress.../lz4-read...] ~/work/compression/rANS_static4 -o0 < /tmp/en9.lz4 | wc -c
Took 2711335 microseconds, 187.1 MB/s
435639681
jkb@seq3a[compress.../lz4-read...] ~/work/compression/rANS_static4 -o1 < /tmp/en9.lz4 | wc -c
Took 6112700 microseconds, 83.0 MB/s
424334289

(The lz4 is an old copy, so I've no idea how it compares speed wise since I downloaded it.)

Anyway the short answer is around 3-4% saving only going from O(0) to O(1) entropy encoding. I expect there are FAR better ways to do this though.

Some timings:
lz4 alone: 4.31sec enc, 0.93sec dec.
lz4 + rans_O(0): 4.45sec enc, 2.76sec dec
lz4 + rans_O(1): 6.38sec enc, 6.94sec dec

So it's still a major hit for decoding performance. Timings here are lz4 | mbuffer | rans or rans | mbuffer | lz4. The mbuffer permits it to use a 2Mb (default) memory buffer to streamline the unix pipes and improve parallel performance. This order-0 decoder is only going at ~240 MB/sec though so FSE or one of Ryg's 64-bit variants would run faster. The only reason I stuck with this version for my work is I wanted something that was identical between 32-bit and 64-bit platforms and it was Good Enough(TM) for the intended use.

James

PS. The rANS_static4 source is ftp://ftp.sanger.ac.uk/pub/users/jkb/rANS_static4.c and based on Ryg's implementation with more aggressive unrolling.

[Sportman]

Input:
691,759,309 bytes, unsorted list with almost 11mln URLs (30% unique).

Output:
253,831,058 bytes, 1.762 sec. - 1.472 sec., tor64 -1
244,090,127 bytes, 1.471 sec. - 0.975 sec., snzip/snunzip (snappy)
229,231,586 bytes, 1.756 sec. - 1.359 sec., qpress64 -L1T1
228,731,049 bytes, 0.344 sec. - 0.340 sec., lz4 -1
225,591,975 bytes, 1.703 sec. - 0.413 sec., lzturbo -10 -p1
213,704,697 bytes, 2.891 sec. - 1.389 sec., qpress64 -L2T1
209,667,579 bytes, 2.638 sec. - 0.553 sec., lzturbo -11 -p1
208,379,530 bytes, 2.166 sec. - 1.341 sec., tor64 -2
204,277,427 bytes, 7.560 sec. - 1.068 sec., qpress64 -L3T1
202,709,615 bytes, 6.581 sec. - 0.553 sec., lzturbo -12 -p1
199,042,115 bytes, 1.594 sec. - 0.612 sec., lzturbo -20 -p1
187,260,122 bytes, 2.097 sec. - 0.330 sec., lz4 -9
185,809,810 bytes, 515.495 sec. - 0.610 sec., lzturbo -19 -p1
166,478,661 bytes, 3.432 sec. - 2.308 sec., tor64 -3
159,816,868 bytes, 2.293 sec. - 0.976 sec., lzturbo -30 -p1
153,453,669 bytes, 9.634 sec. - 4.047 sec., zhuff_beta -c1 -t1
148,006,274 bytes, 14.177 sec. - 0.628 sec., zhuff_beta -c2 -t1
145,879,704 bytes, 2.066 sec. - 0.610 sec., zhuff_beta -c0 -t1
139,594,362 bytes, 1.970 sec. - 0.665 sec., lzturbo -21 -p1
132,484,415 bytes, 3.446 sec. - 1.938 sec., tor64 -4
113,994,789 bytes, 5.757 sec. - 0.631 sec., lzturbo -22 -p1
111,101,281 bytes, 2.353 sec. - 0.930 sec., lzturbo -31 -p1
100,787,832 bytes, 6.946 sec. - 2.280 sec., tor64 -5
84,223,767 bytes, 6.036 sec. - 0.849 sec., lzturbo -32 -p1
80,364,741 bytes, 9.534 sec. - 1.928 sec., tor64 -6
76,132,986 bytes, 23.006 sec. - 1.901 sec., tor64 -11
70,167,371 bytes, 34.920 sec. - 1.800 sec., tor64 -12
69,537,310 bytes, 20.230 sec. - 1.763 sec., tor64 -7
67,239,044 bytes, 56.043 sec. - 1.747 sec., tor64 -13
65,988,172 bytes, 69.410 sec. - 1.754 sec., tor64 -14
64,155,952 bytes, 119.300 sec. - 1.721 sec., tor64 -15
63,841,180 bytes, 32.758 sec. - 1.767 sec., tor64 -8
62,863,233 bytes, 394.516 sec. - 0.681 sec., lzturbo -29 -p1
62,267,140 bytes, 51.723 sec. - 1.764 sec., tor64 -9
61,110,352 bytes, 61.808 sec. - 1.769 sec., tor64 -10
60,187,593 bytes, 260.807 sec. - 1.665 sec., tor64 -16
51,475,718 bytes, 456.172 sec. - 0.844 sec., lzturbo -39 -p1
48,497,129 bytes, 654.649 sec. - 2.716 sec., lzturbo -49 -p1

[fcorbelli]

stdin-stdout does not seems to work fine on Windows.

something like that

mysqldump -uroot -ppassword database | gzip >1.gz

is OK
but

mysqldump -uroot -ppassword database | lzturbo -of -10 >1.lzt

throw an error

mysqldump: Got errno 22 on write

[Sportman]

@avita:
i'm considering making a 32 bits in the near future.

I use LzTurbo to compress one kind of data where it's better then any other kind of compressor in speed enc/dec/size combination. Problem I face is that sometimes I need to run it on 32-bits Win OS and need to switch back to an other compressor I used before, what is much less optimal and use double time enc/dec and create a bigger output. Any change to compile a 32-bits version of LzTurbo, I use only -32 -b1024 -p1 to compress and -d -p1 to decompress?

[a902cd23]

It looks like LzTurbo cannot handle files opened by other files, if I open a DVD-movie (ISO-file) with VLC and then try to compress it I only get a 32bytes file. After closing VLC LzTurbo can handle file.

[dnd]

@Sportman: lookt at PM

@a902cd23: thanks for reporting. I'll look at it in the next LzTurbo update