LZTURBO 0.91 Parallel Compressor (Win32/Linux)

[donotdisturb]

New version & Benchmarks: lzturbo 0.91 parallel compressor
at: http://consultant-berater.de/lzturbo

- Compress better & faster (Levels 1..
- (Level 9 minor changes in speed)

Please compare lzturbo to thor, tornado, slug, quicklz. Specially on newer hardware (like core 2 duo).
------- SINGLE CORE WAS YESTERDAY -------

Please read lzturbo.txt.
Please read the benchmarks at:
http://consultant-berater.de/lzturbo for the proper switches to choose.

Enjoy!

[LovePimple]

Thanks Hamid!

[LovePimple]

Quick (-59) test...

A10.jpg > 839,071
AcroRd32.exe > 1,417,442
english.dic > 687,644
FlashMX.pdf > 3,736,621
FP.LOG > 789,021
MSO97.DLL > 1,810,448
ohs.doc > 833,243
rafale.bmp > 1,006,476
vcfiu.hlp > 677,425
world95.txt > 576,470

Total = 12,373,861 bytes

[Nania Francesco]

MOC Test
option -59 -> 159.922.717 b comp. 465,715 s. - dec. 40,782 s.

[LovePimple]

Another quick test...

Test machine: Intel PIII (Coppermine) @750 MHz, 512 MB RAM, Windows 2000 Pro SP4

Test File: ENWIK9 (1,000,000,000 bytes)

Timed with AcuTimer v1.2


Compression settings: -11

Compressed size: 430,453,227 bytes
Elapsed Time: 00:02:05.594 (125.594 Seconds)


Compression settings: -12

Compressed size: 375,431,333 bytes
Elapsed Time: 00:02:40.155 (160.155 Seconds)

[maadjordan]

great work..10x thanks

[donotdisturb]

Thanks LovePimple, Nania & Maadjordan.

Updated new version & benchmarks: lzturbo 0.92
at: http://consultant-berater.de/lzturbo
(Level 9 unchanged).

Nania Francesco Antonio
Thanks for testing. The decompressor from 0.9 code is unchanged
in this version, but you are reporting other timings in your quick-test (40s for 0.91 versus 7s for 0.9).

[LovePimple]

Thanks Hamid!

[Nania Francesco]

Unfortunately this program had given me problems with the test and evidently has to be us an error in the precedents effected test! Have had to rewrite the code of the program of test for every type of compressor! I have referred the MOC test and the result it is the same.

[Vacon]

Hello everyone,

------- SINGLE CORE WAS YESTERDAY -------

Suddenly I feel sooo old...
Do I have to bury my AMD 1700+ now?


Best regards though!

[encode]

------- SINGLE CORE WAS YESTERDAY -------

Not sure bout that. I think, mainly, all your noise about multi-core systems is no more than just LZTURBO advertisment. The power of dual/quad/... core systems applying to the data compression is overestimated, generally speaking. Maybe the coolest part is that you can compress something and at the same time do something another with minor performance loss - i.e. browsing the web, whatching movies... Using two or more cores not means that youll getting 2X-4X... speed boost - as you know, and especially for LZ-based coders, the main bottleneck is the memory access due to string searching. So LZ77 benefits more from faster memory/cache - not faster CPU. With LZTURBO, I suggest, instead of a TORNADO clone a la "lots of stuff" inside, make some LZMA clone with greater performance - i.e. higher compression/faster decompression... It whould be interesting - since the main purpose of any experimental compressor is to introduce something new, new idea, new approach, new algorithm...
Recently, got a new GloFiish SmartPhone/Communicator (Or how to call the unit which has ALL stuff, including GPS,Wi-Fi,Camera,even FM-transmitter... Crazy device). And saw how efficient Deflate (ZIP) is, under Windows Mobile 6. Hm, maybe Ill write something BALZ-based for WM... PocketBALZ, BALZ in the Pocket, or whatsoever... PocketLZ... The first experimental file compressor for Pocket PC. Sounds great! Does anyone have SmartPhone/PocketPC/... just to see, are anyone able to test the stuff... You know what Im sayin...

[Simon Berger]

@encode
I?m irritated to read such a post from a developer of many compression programs and also how you write it. There aren?t many programs availible yet, taking advantage of multicore systems thats right, but if you see 7zip for example it is much faster using threads there. With more and more cores the advantage isn?t so big and also the hard drive could be a second slow down (but raid systems going to be common more and more which can speed up the access).
At the moment the multicore systems almost only have the advantage you mentioned (several processes) but on XP this doesn?t work as good as it should (own experience).

I thank donotdisturb to take this step because most developers only seem to be to lazy to implement a thread algorithm. Don?t you think it could speed up the slow compression time of balz?

[encode]

I just have my own opinion, which can be straight - just as I think. I hope youre correctly understand my position.

Don?t you think it could speed up the slow compression time of balz?

As I noted, mostly, the time taken by memory access, that explains the great difference in performance on my and Matts (LTCB) PC. I may greatly improve the compression speed, switching match finder to binary trees, instead of a hash chains, someone may say that using hash chains with matching from each byte is lame - at some point, yes it is. Hash chains are very efficient with fast, unoptimized parsing - with greedy/lazy matching. However, currently I use this approach, even I know that it isnt optimal solution.
Later will play with Multi-Threaded things, but as I posted, MTs benefits are overestimated by the users, I just understand there is no magic. Multi PC compression is more advanced idea - combining the power of a few PCs together, syncronizing it via USB/LAN/Wi-Fi, since each unit has independed memory/cache and CPU, we may achive some interesting performance... And here, two cores but physically single PC - one motherboard, one or two memory channel(s), etc. Im very sceptic.

[Black_Fox1]

all your noise about multi-core systems is no more than just LZTURBO advertisment

HW manufacturers produce quad-core CPUs as a mainstream (and already there are plans for hexa-core at Intel and octal-core at AMD) and even GPUs are using higher number of chips, but I bet you are aware of that and just mean another thing: Programs performance cannot magically scale just because more threads are available, there have to be some changes on the code part. Games will be paralelized in some years dramatically, as CPUs/GPUs will be able to render raytracing in real-time, some programs also as well. However, there is little to speed up in compression world - only thing, that can be done, is splitting input into more parts, which degrades ratio Is there any theoretical chance, that (almost perfectly) scalable algorithm will be ever found?

[Sportman]

Microsoft is busy to make it more easy to program multi core in .NET Framework, I didn't test it yet:

http://msdn2.microsoft.com/en-us/magazine/cc163340.aspx
http://blogs.msdn.com/pfxteam/archiv...9/6558543.aspx
http://www.microsoft.com/downloads/d...E-B204472838E1

http://www.microsoft.com/downloads/d...5-024bc7f180ba

 
4Gb Game (45653 files) 
Core2 1.67 , DDR2-666 2Gb 
 
Ratio   //   Comp.   //  Decomp.          // Archiver 
35.043% //  1420kb/s // 18095kb/s(no out) // 7-zip 4.56 Ultra -d110m 
43.240% //  6407kb/s // 17929kb/s         // Tornado 0.3 -6 (stored 7z) 
43.629% //  8059kb/s // 18179kb/s         // Tornado 0.3 -5 (stored 7z) 
44.223% //  8387kb/s // 16561kb/s         // Lzturbo 0.92 -44 (stored 7z) 
45.026% //  4587kb/s //  8590kb/s         // Lzturbo 0.92 -55 (stored 7z) 
46.404% // 12942kb/s // 18653kb/s         // Tornado 0.3 -4 (stored 7z) 
46.631% // 10205kb/s // 16053kb/s         // WinRar v3.71 Fastest (stored 7z) 
50.938% // 17449kb/s // 19740kb/s         // Tornado 0.3 -3 (stored 7z)

Hdd limit is somewhere ~17mb/s so keep in mind.

[encode]

HW manufacturers produce quad-core CPUs as a mainstream (and already there are plans for hexa-core at Intel and octal-core at AMD)

Manufacturers just desperate to grab the last cent from our pocket...

[Bulat Ziganshin]

Zonder
for tornado yoo can use -o option (w/o parameter) to (de)compress into null

[Bulat Ziganshin]

There aren?t many programs availible yet, taking advantage of multicore systems thats right

i wonder whether you really use any single-file compressors? my opinion is that programs published here are mostly research in compression technologies rather than really useful products. they usually dont include any multithreading technologies exactly because its trivial to implement. imho, for real usefulness the program should be either an archiver or compressor that supports linux/windows, crc checking and stdin-to-stdout mode

what are opinions of other readers? does such programs have real applications? if you use/want to use standalone compressors, what you need from such programs?

[Simon Berger]

i wonder whether you really use any single-file compressors? my opinion is that programs published here are mostly research in compression technologies rather than really useful products.

You are right on this point but I will ask you a counter question. Do you try to create a compression engine which isn?t competitive and slow at all?
Multi threading techniques are the future (what we can foresee from now) because there going to come many more cores. I also don?t like programming with threads but I learn and do it because it?s not trivial to implement them and you will need them more and more because the market will ask more and more
I don?t see your point why a "fun project" and research shouldn?t include multi threading. It isn?t trivial to change a algorithm to multi threading and sometimes impossible (to get a good from it).

[Bulat Ziganshin]

Do you try to create a compression engine which isn?t competitive and slow at all?

try to compare lzturbo and tornado pure compression power. force lzturbo to single-thread mode and look at "user time" printed by Timer

I also don?t like programming with threads but I learn and do it because it?s not trivial to implement them

its easy, you just need to use proper tools/techniques. freearc runs multiple threads just by using unix-style notation:

reader |> filter |> filter |> compressor |> writer

as i already said, we dont add MT techniques to our fun projects exactly because its trivial to compress multiple data blocks in parallel. OTOH, lzturbo is aimed to practical usage and its natural to add all those bells and whistles here. it will be even better to provide MT environment just around proven tornado core, but i cant force him to do it

tor0.1 split data into independent chunks and compressed them separately. it probably becomes source of inspiration for Hamid who added MT features to my core. lzturbo 0.91 was probably
upgraded to tor03 core but still uses independent chunks

OTOH, implementing multicore compression *engine* isnt trivial task. its why they are so rare and of course we all agree that such researches will be highly useful

[Bulat Ziganshin]

i need to add that only invention of fast and memory-efficient match finder in tornado 0.1 made it possible lzturbo to appear. if it used lzma compression engine, its memory reqs will be 6-8x larger!!!

[Black_Fox1]

I think there's some suboptimality in LZTurbo's optimal parsing. I have a tar archive, which is very compressible (7z compresses it from 63 MB to 426 kB)... it took 3 hours to compress with PAQ8o9 (using one core) and 2 hours for latest LZTurbo using both cores...

[Bulat Ziganshin]

i'm pretty sure that reason is the naive binary tree implementation which makes lzturbo extremely slow on any binary data. try cabarc on this file - it was also naive otoh, Hamid optimizes lzt for enwik9, so i don't expect that he will make any changes for other datatypes

[Black_Fox1]

Good idea, but that's not it - cabarc finished in about a minute. Maybe the difference comes from dictionary size.

I made just a simple test - some results are here and the 7zipped file is here
saves.tar is original file, saves.rep is saves.tar processed with rep

[Bulat Ziganshin]

well, another idea: lzt uses hc or ht with a very deep search (say, 1000 elements) what makes it very slow when there are lot of 4-byte repetitions

[encode]

hc or ht with a very deep search (say, 1000 elements)

1000 is not so deep. All versions of BALZ uses HC with 4096 "elements"... as Deflate does - with highest compression it restricts the chain length to 4096.