LZTURBO 0.0.1 - Experimental version

[donotdisturb]

Hi,
lzturbo is an lz77 based multithreaded and linear scalable (in multicore systems) file compression program using a fast state-of-the-art optimal parsing (option -49) and a fast arithmetic coder with very fast decompression.
Please compare with quicklz (option -11) or other compression-programs (option -49).

There is no compressor that compresses better with faster decompression (option -49).

download (at this time, only linux version):
http://www.consultant-berater.de/lzturbo-0.0.1.tar .gz

usage:
lzturbo -1X input output
lzturbo -2X input output
lzturbo -4X input output
X=1..9

ex. compression:
lzturbo -49 enwik9 enwik9.lz

decompression:
lzturbo -d enwik9.lz enwik9.lzd

Enjoy!

[nimdamsk]

Thanks!
Waiting for Win32 version

[Vacon]

Hello everyone,

Waiting for Win32 version

Me too
Seems to be interesting!
License? Source? (Linux does not necessarily mean GPL...)

Best regards!

[encode]

There is no compressor that compresses better with faster decompression (option -49).

I doubt it. A revolution form noname? A bluff!

[Vacon]

Hello everyone,

There is no compressor that compresses better with faster decompression (option -49).

... doesnt mean much at all, _except_ that -> for the decompression time LZTURBO 0.0.1 needs there is no other program with better compressing.
So _I_ read -> decompression time is quick. Not more, but even not less!
Cmon, give him a chance
Im waiting for first results.

Best regards!

[nimdamsk]

Who knew ccm and slug author several months ago?

[Vacon]

Hello everyone,

Who knew ccm and slug author several months ago?

And who wants to miss them now...?
Still waiting for results.

Best regards!

[encode]

Who knew ccm and slug author several months ago?

Unique case... By the way, looks like Chris waiting in shadows. Hes disappeared. As just as DARKs author! -- catch me if you can, you know...

[Vacon]

Hello everyone,

-- catch me if you can, you know...

Yeah, sometimes author play "hit and run"

Best regards!

[Black_Fox1]

Thanks donotdisturb!

Could someone provide any test numbers/results?

[Matt Mahoney]

I tried testing under Ubuntu but could not get it to work.
Perhaps the problem is I can't link a 64-bit shared library to a 32-bit executable? I might get it to work with source code, or a Win32 exe.

matt@matt-desktop:~/Desktop$ ./lzturbo
./lzturbo: error while loading shared libraries: libmagic.so.1: cannot open shared object file: No such file or directory

matt@matt-desktop:~/Desktop$ file /usr/lib/libmagic.so.1
/usr/lib/libmagic.so.1: symbolic link to 'libmagic.so.1.0.0'

matt@matt-desktop:~/Desktop$ file /usr/lib/libmagic.so.1.0.0
/usr/lib/libmagic.so.1.0.0: ELF 64-bit LSB shared object, AMD x86-64, version 1 (SYSV), stripped

matt@matt-desktop:~/Desktop$ file lzturbo
lzturbo: ELF 32-bit LSB executable, Intel 80386, version 1 (GNU/Linux), statically linked, stripped

matt@matt-desktop:~/Desktop$ uname -a
Linux matt-desktop 2.6.15-28-amd64-generic #1 SMP PREEMPT Thu May 10 09:46:40 UTC 2007 x86_64 GNU/Linux

[encode]

./lzturbo: error while loading shared libraries: libmagic.so.1: cannot open shared object file: No such file or directory

Same thing with Slax LiveCD...

[Bulat Ziganshin]

I doubt it. A revolution form noname? A bluff!

there are only two lz77 compressors with optimal parsing - cab and lzma. cab has very fast decompression but pretty simple model and only 2mb dictionary. otoh, lzma has sophisticated model which leads to several times slower decompression. i guess that lzturbo lies between these two programs in terms of both speed and compression

imho, it should be possible to construct such algorithm from tornado sources (which provides lzari model) and lzma sources (which provides optimal parser)

many years ago i "reimplemnted" arj algorithm by just combining zip match finder with ar002 model

on my box (fedora 7/32 bit) i was able to run 1* and 2* models, but not 4*. -11 is only 2x faster than gzip -1 so probably it doesnt have any chances against quicklz

[encode]

Okay, some first digits (-49 mode):

world95.txt: 581,517 bytes

fp.log: 794,437 bytes

ENWIK8: 26,678,709 bytes

calgary.tar: 876,920 bytes

It worked on Fedora 7 KDE Live. Compression is slow even on my machine, sometimes lzturbo even frozes... Decompression is fast indeed.

[Vacon]

Hello everyone,

so here we are -> discussing facts!

Best regards!

[encode]

Yet another results (-49):

A10.jpg: 839,822 bytes
acrord32.exe: 1,429,364 bytes
english.dic: 792,403 bytes
FlashMX.pdf: 3,741,354 bytes
fp.log: 794,437 bytes
mso97.dll: 1,827,499 bytes
ohs.doc: 838,887 bytes
rafale.bmp: 1,026,827 bytes
vcfiu.hlp: 685,044 bytes
world95.txt: 581,517 bytes
Totally: 12,557,154 bytes

pht.psd: stack overflow. Aborted

pak0.pak: frozes

PariahInterface.utx: frozes

It uses up to 764 MB for compression. On some files it either crashes or freezes. In other words, it's buggy...

[encode]

It's interesting to see the program's details (algo/implementation details). Also waiting for WIN32 release. (Maybe all bugs caused by Linux incompatibility...)

[donotdisturb]

NEW:
download linux version (with static-linked libmagic +libpthread)
+win32-exe (win32-version tested only on linux!).

http://www.consultant-berater.de/lzturbo-0.0.1.tar .gz

Please read the file lzturbo.txt.
Please use the linux version when possible.

Enjoy!

[donotdisturb]

sorry!
http://www.consultant-berater.de/lzturbo-0.0.1.tar .gz

[Black_Fox1]

Thanks donotdisturb!

results on my testset (-49 parameter):
(my 6 755 163 bytes PSD file produced stack overflow error and wasn't counted to stats)

original size: 23 554 031
compressed:13 227 143
comp speed: 729 kB/s
dcmp speed: 3 795 kB/s

[Lasse Reinhold]

Hi donotdisturb,

How about adding a -mem flag for benchmarking? Feel free to use the code from the quicklz demo

Lasse

[LovePimple]

Thanks donotdisturb!

[Matt Mahoney]

It works in Windows XP on enwik8/9 with one minor problem. The output files are read-only and have to be changed with "attrib -r" before you can modify or delete them. I didn't test the Linux version.
http://cs.fit.edu/~mmahoney/compression/text.html# 2333

It also beats lzpm for decompression speed and size.

[LovePimple]

Quick test with Win32 version...

Test machine: AMD Sempron 2400+, WinXP SP2


ENWIK8 (-11)
Compression Time: 3.045 Seconds
Compressed Size: 55,397,274 bytes

ENWIK8 (-21)
Compression Time: 3.298 Seconds
Compressed Size: 52,727,086 bytes

ENWIK8 (-22)
Compression Time: 6.505 Seconds
Compressed Size: 46,376,388 bytes

ENWIK8 (-23)
Compression Time: 13.229 Seconds
Compressed Size: 39,822,079 bytes

ENWIK8 (-24)
Compression Time: 86.941 Seconds
Compressed Size: 35,169,367 bytes

ENWIK8 (-49)
Compression Time: 312.522 Seconds
Compressed Size: 26,678,709 bytes

ENWIK8 (-19)
Compression Time: 541.577 Seconds
Compressed Size: 36,892,158 bytes

ENWIK8 (-29)
Compression Time: 542.872 Seconds
Compressed Size: 33,883,186 bytes

[encode]

It also beats lzpm for decompression speed and size

Cool! (Dont forget to check out the LZPM 0.09)

LZTUTBO is very efficient on text files, however, check out the decompression speed at Black_Foxs test set:

LZTURBO:
comp speed: 729 kB/s
dcmp speed: 3 795 kB/s

LZPM:
comp speed: 1 028 kB/s
dcmp speed: 9 866 kB/s

Different picture, isnt it? LZPM decompresses nearly 3X times faster.

Possibly, LZTURBO was preferable tested and tuned on ENWIK8/9.

By the way, Hamid, can you provide us some info on your compression (preferable, on -49 mode):

Arithmetic coding, OK. Bit-oriented/byte oriented? Pure order-0/order-1 or order-1-0 PPM, or order-1-0 CM, or?

Super-optimal parsing? OK. What this means? LZMA/CABARC-like? Does it estimate cost for each choice, or it uses constants, and so on...

Match finder. What kind of. Hash chains? Binary tree?

Is LZTURBO pure LZ77, or it uses some tricks, e.g. ROLZ, ... ?

Also, its might be interesting for users and me, can you introduce yourself? For example, do you enthusiast, or maybe youre vice president of Microsoft, etc.

Sorry for bugging you, just this forum and compression community welcomes a new member!

[LovePimple]

Another quick test...

PariahInterface.utx (-49)

Compressed Size: 5,555,230 bytes
Compression Time: 3094.160 Seconds
Decompression Time: 7.053 Seconds


PHT (-29)

Compressed Size: 1,397,693 bytes
Compression Time: 8.430 Seconds
Decompression Time: 0.320 Seconds


Scribble.wav (-49)

Compressed Size: 17,708 bytes
Compression Time: 109.376 Seconds
Decompression Time: 0.348 Seconds

[donotdisturb]

Hi,
lzturbo is designed as a practical compressor for modern processors not for benchmarking.
There is no sense to test the speed for small files, as it's not important if a user wait's 3 or 6 seconds, but there is a big difference (from the user view), between 10min or 35min. Please compare speeds only for equivalent compression ratios. Look also at the times 'real' (times users waiting for, including I/O).
Note: This is only experimental version 0.0.1

Now the results on my system: LZTURBO 0.0.1
Hardware: AMD-X2-3800+ dual core. Ubuntu-Linux 7.04 32-bits:
enwik9:
file copy: cp enwik9 enwik9.lz real:37 user:0.12

lzturbo using 2 cores (note:user time per core).
lzturbo mode -11: 493689235 bytes
comp: real:12.7 user:3.3 (per core) decomp:real 17.5 user:1.6 (per core)

lzturbo mode -13: 378029486 bytes
comp: real:21.3 user:16.4 decomp: real:19.7 user:2.3

lzturbo mode -49 233322999 bytes
comp: real:637 user:600 decomp: real:32 user:22

lzpm 0.0.9: 242929442
comp: real:2017 user:2002 decomp:real:66 user:53

quicklz 1.30 final:
mode -0: 469460173 bytes
comp: real:22.5 user:7.7 decomp:real 20.1 user:5.9

mode -3:411462500 bytes
comp: real:54 user:39.2 decomp:real:18.9 user:3.8

[squxe]

wait for win32 version
sorry it's here
http://www.consultant-berater.de/lzturbo-0.0.1.tar .gz

[LovePimple]

There is no sense to test the speed for small files,

The smallest file in my quick test was PHT.psd at 6.44 MB (6,755,163 bytes).

Scribble.wav = 8.82 MB (9,249,480 bytes)

PariahInterface.utx = 23.2 MB (24,375,895 bytes)

Look also at the times real (times users waiting for, including I/O).

All of my tests include I/O time.

Note: This is only experimental version 0.0.1

Looking forward to future releases!

[Bulat Ziganshin]

Possibly, LZTURBO was preferable tested and tuned on ENWIK8/9.

look at the following:

1:byte-output (fastest decompression)
2:bit-output
3:not implemented
4:best compression

it looks like he just used my tornado model. in this case i know why its so bad on non-text files - tornado 0.1 MF doesnt find 2/3-byte strings nor contains many other improvements which improves binary files compression in rar/cab/lzma. since 0.1 ive improved binary files comporession by at least 10%