LZTURBO 0.9 parallel compressor

[donotdisturb]

New version : lzturbo 0.9 parallel compressor
at http://consultant-berater.de/lzturbo

- improved & faster compression.
- compress/decompress multiple files in parallel.
- compress/decompress directories recursively.
- compress/decompress as easy as the copy command. No archive files needed.

Please compare lzturbo to thor, slug, quicklz. Specially on newer hardware (like core 2 duo).
Use switches as recommended in lzturbo.txt file.

Please read lzturbo.txt

Enjoy!

[LovePimple]

Thanks Hamid!

[LovePimple]

Quick test...

Compression settings: -11

A10.jpg > 842,504
AcroRd32.exe > 2,095,707
english.dic > 1,648,598
FlashMX.pdf > 4,341,107
FP.LOG > 2,032,838
MSO97.DLL > 2,649,965
ohs.doc > 1,107,412
rafale.bmp > 1,912,757
vcfiu.hlp > 1,142,726
world95.txt > 1,243,243

Total = 19,016,857 bytes


Compression settings: -59

A10.jpg > 839,071
AcroRd32.exe > 1,417,442
english.dic > 687,644
FlashMX.pdf > 3,736,621
FP.LOG > 789,021
MSO97.DLL > 1,810,448
ohs.doc > 833,243
rafale.bmp > 1,006,476
vcfiu.hlp > 677,425
world95.txt > 576,470

Total = 12,373,861 bytes

[LovePimple]

Another quick test...

Test machine: Intel PIII (Coppermine) @750 MHz, 512 MB RAM, Windows 2000 Pro SP4

Test File: ENWIK9 (1,000,000,000 bytes)

Timed with AcuTimer v1.2


Compression settings: -11

Compressed size: 430,453,227 bytes
Elapsed Time: 00:02:05.970 (125.970 Seconds)


Compression settings: -12

Compressed size: 386,091,908 bytes
Elapsed Time: 00:02:49.249 (169.249 Seconds)

[donotdisturb]

Thanks LovePimple!

[Matt Mahoney]

http://cs.fit.edu/~mmahoney/compression/text.html#2328

[thometal]

Quick Test:

Eve Online installation:

uncompressed 3.037.826.415 Bytes

7z -md=128m -mfb=273 -ms=on -mx=9
compressed 1.204.469.019 Bytes
Kernel Time = 20.546 = 00:00:20.546 = 0%
User Time = 4742.406 = 01:19:02.406 = 160%
Process Time = 4762.953 = 01:19:22.953 = 161%
Global Time = 2946.204 = 00:49:06.204 = 100%

LZTurbo -59 -r
compressed 1.427.313.134 Bytes
Kernel Time = 64.968 = 00:01:04.968 = 0%
User Time = 13569.484 = 03:46:09.484 = 198%
Process Time = 13634.453 = 03:47:14.453 = 199%
Global Time = 6832.703 = 01:53:52.703 = 100%

[donotdisturb]

Thanks Matt!
Thanks thometal !

thometal
Can you report decompression time, please.

[thometal]

7z decompressions time:
Kernel Time = 8.843 = 00:00:08.843 = 2%
User Time = 194.953 = 00:03:14.953 = 62%
Process Time = 203.796 = 00:03:23.796 = 65%
Global Time = 312.359 = 00:05:12.359 = 100%

LZTurbo will i post later

[thometal]

LZTurbo decompressions time:

Kernel Time = 22.640 = 00:00:22.640 = 5%
User Time = 586.765 = 00:09:46.765 = 143%
Process Time = 609.406 = 00:10:09.406 = 148%
Global Time = 409.657 = 00:06:49.657 = 100%

[donotdisturb]

enwik9: AMD X2-3800+ (Dual core)
__________________________size____Global comp._time in sec.
7z_-md=128m_-mfb=273___206938137__32m13
___-ms=on_-mx=9
lzturbo_______-59________232701587___9m33

thor_0.96a_e4___________314092324___1m05
thor_0.96a_e5___________315611172___1m27
lzturbo_______-43________318502927_____35
gzip_-9_________________322591995___1m41

lzturbo_______-41________361432918_____24.82
quicklz1.4b6_3___________397439311___1m20

lzturbo_______-21________401221587_____25.37
slug1.1_________________404250979_____26.0
quicklz1.4b6_2___________406053574_____30.38
lzturbo_______-11________430453227_____30.52
quicklz1.4b6_1___________466309381_____32.4
thor_0.96a_e1___________488397982_____32.6

[Piotr Tarsa]

donotdisturb
use CODE tag

[donotdisturb]

 
enwik9: AMD X2-3800+ (Dual core) 
                          size    Global comp. time in sec. 
7z -md=128m -mfb=273   206938137  32m13 
   -ms=on -mx=9 
lzturbo       -59      232701587   9m33 
lzturbo	      -44      297641351   1m21 
 
thor 0.96a e4          314092324   1m05 
thor 0.96a e5          315611172   1m27 
lzturbo       -43      318502927     35 
gzip -9                322591995   1m41 
 
lzturbo       -41      361432918     24.82 
quicklz1.4b6 3         397439311   1m20 
 
lzturbo       -21      401221587     25.37 
slug1.1                404250979     26.0 
quicklz1.4b6 2         406053574     30.38 
lzturbo       -11      430453227     30.52 
quicklz1.4b6 1         466309381     32.4 
thor 0.96a e1          488397982     32.6

[encode]

donotdisturb
By the way, can you describe the -59 algorithm? How do you search matches (hash chains, binary tree, ...), how do you encode LZ output, and so on.

[Piotr Tarsa]

i'm curious about 7- zip resluts in lzma mode with 1 gb dictionary

[LovePimple]

The MC SFC summary shows a total of 12,381,859 bytes rather than the 12,373,861 bytes I got using the -59 setting.

http://www.maximumcompression.com/data/summary_sf.php

[Bulat Ziganshin]

how do you encode LZ output, and so on

it seems that he used coders from tornado for -1*/-2*/-4* modes

[encode]

it seems that he used coders from tornado for -1*/-2*/-4* modes

Indeed suspicious compressor - too many mirrored features from TORNADO...

[Bulat Ziganshin]

i don't think that there are many people who can develop good huf/ari encoder, so it's not surprising