LzTurbo Byte coding vs. Zhuff FSE

[dnd]

LzTurbo with byte coding beats Zhuff with Huffman/FSE
Decompression is 4 to more than 12 times faster.

Single core in memory benchmarks with i7-2600K CPU at 3.4GHz.
Corpus from: http://compressionratings.com/

                    size     ratio    C MB/s     D MB/s     
app1.tar          141049191  28.0     316.40    2210.77    lzturbo 21 
app3.tar           52026869  52.0     231.39    1817.04    lzturbo 21 
app4.tar          129543129  31.7     239.01    1730.35    lzturbo 21 
audio1.tar        105295850  99.4     414.27    4952.80    lzturbo 21
cr_txt1            46960579  47.0     152.80    1204.00    lzturbo 21 
enwik8             42537930  42.5     157.94    1166.46    lzturbo 21 
game1.tar         280591204  40.0     208.73    1668.39    lzturbo 21
game2.tar         265484767  43.8     248.74    1970.52    lzturbo 21
img1.tar           64282288  70.6     170.99    1887.94    lzturbo 21
src1.tar           26717627  26.4     253.91    1712.09    lzturbo 21
ubuntu500m.tar    163795024  32.8     227.52    1665.55    lzturbo 21    
Total:           1318284458


app1.tar          167730707  33.3     213.4      460.3     zhuff v0.9c
app3.tar           53083567  53.0     163.2      345.0     zhuff v0.9c
app4.tar          154295202  37.8     171.9      235.4     zhuff v0.9c
audio1.tar         98528678  93.1     167.3      271.1     zhuff v0.9c
cr_txt1            43099756  43.1     138.1      270.5     zhuff v0.9c
enwik8             40643947  40.6     145.5      293.1     zhuff v0.9c
game1.tar         276499255  39.5     171.8      353.1     zhuff v0.9c
game2.tar         251632184  41.5     178.9      388.2     zhuff v0.9c
img1.tar           54985329  60.4     122.3      302.2     zhuff v0.9c
src1.tar           28318301  28.0     208.7      401.3     zhuff v0.9c
ubuntu500m.tar    178265751  35.7     180.8      150.2     zhuff v0.9c
Total:           1347082677

[Piotr Tarsa]

Does lzturbo use any preprocessors/ filters/ special models?

[Bulat Ziganshin]

lzturbo uses STOLEN tornado codebase plus its own, buggy and slow optimal parser. afair, it was up to 100 times slower than original code in independent tests

as usual, hamidi compares apples to oranges - zhuff employs only one thread. as one can see in the CR benchmarks, 4x4 or nanozip easily outperform that piece of stolen code

[m^2]

Bulat, he did not steal it. At most, he did not comply with your license.

dnd, what lzturbo is this? Can you provide a link to download, so your claim can be independently verified?

[Bulat Ziganshin]

m^2, what is the difference?

hamidi removes previous versions of his program and prohibits anonymous downloads in order to complicate analysis of stolen things. but as we previously found, he uses matchfinders and entropy coders from my program, up to exact contents of len/dist encoding tables

[dnd]

- Does lzturbo use any preprocessors/ filters/ special models?
Only e8e9 filter for ".exe", ".dll" and ".tar" files.

- link to download LzTurbo: https://sites.google.com/site/powturbo/

I don't want to start again the eternal discussion. Everybody can see from the benchmarks that Lzturbo has nothing to do with Tornado.
Bulat, please go watch TV or go search for a girl friend.

[m^2]

Bulat, what's the similarity?

Come on, these 2 things are extremely different. To steal is to take away. There's loss. That's much more than to act against one's will even if in most countries, civil law gives this will some extraordinary power.

[Bulat Ziganshin]

it's not the discussion - that's the OBVIOUS thing. the first version of lzturbo has EVERYTHING copied from the tornado 0.1 plus the optimal parser. It was buggy and slow, are you remember? It shows your programming level. Noone understanding LZ can believe that you have developed exact copy of tornado but was unable to provide optimal parser of reasonable quality. later you continued to copy code from new versions of tornado, releasing new lzturbo versions in a few days after tornado releases. it is how lzturbo got huffman encoder, for example

- link to download LzTurbo: https://sites.google.com/site/powturbo/

i tried several times and never succeed. that shows the difference - i publish my sources and you prohibit me even to download your compiled code since you fear the disassembly analysis

[Bulat Ziganshin]

m^2, without idea of intellectual property you will never see the films, books or software except those written and filmed by enthusiasts. stealing IP made me lost money that i can got by licensing the tornado sources

[thometal]

hm but i can download lzturbo, neednt to register

[Black_Fox]

Attaching the only download that is available from the site for anyone who can't download it from the Google pages.

[m^2]

m^2, without idea of intellectual property you will never see the films, books or software except those written and filmed by enthusiasts.

Which is not a problem for me. I avoid others anyway. Which doesn't change the fact that many of those enthusiast actually make money from their creations.

stealing IP made me lost money that i can got by licensing the tornado sources

The following assumes that dnd indeed took your code. I want to make it clear that I'm not sure it's true, though I believe it is.
You seem to assume that if dnd hadn't taken the code that you have written for free, he would have paid for it. Which ignores several other, legal possibilities:
* he would have published lzturbo sources
* we would have taken some other code
* we would have written lzturbo from scratch
* he wouldn't have written lzturbo at all

Reminds me of bankers who just after giving you a credit already lend the profits they are to take.
In case of banks the scale is large enough for statistics to work, so they can model their losses and still it proved to be highly unstable.

[Bulat Ziganshin]

hm but i can download lzturbo, neednt to register

try to sign out of the google account and then download again. it doesn't need to regitser, hamidi just prohibited to download either anonymously or from my account. of course, i can find another one - it just shows how he fear that i will analyze his code

[Bulat Ziganshin]

m^2, in this case you should leave this forum since it was developed by a commercial company and was unable without selling the licenses. and i can't beleive that you really never seen any of Hollywood films

i don't expect that he will pay for anything, i mean that he SELLS my code. and that he make clumsy assistance for all real iranian programmers trying to sell results of their work

[Bulat Ziganshin]

The following assumes that dnd indeed took your code. I want to make it clear that I'm not sure it's true

just recall the first lzturbo version. what was the only difference against tornado 0.1, published a few monthes ago? optimal parser. i have wrote better one in a 3 days, it is much simpler than master a multitude of match finders and encoders. and even that part of lzturbo was slow and buggy, he was fixing it more than a year. i think this tells enough about his programming expertise. nevertheless, the rest of lzturbo code was quick and stable from the first version

at the March 2008 i released tornado 0.4 that was added Huffman coder. In just a week or so it was implemented by lzturbo, and it got two new versions in quick succession although there were no releases in a half-year. once again - he don't get any troubles inplementing Huffman coder while was still uable to debug his 6 monthes-old optimal parser. It shows a clear distinction between his own and stealed code

[dnd]

Bulat, instead of spreading lies please show to the world where is your optimal parser. UNTIL TODAY, THERE IS NO OPTIMAL PARSER IN TORNADO!!!!

I have already published the source code of the huffman coder and the match finders used in the previous versions of lzturbo, but you are still ignoring the reality.

Your Tornado is GPL, so you can't prevent somebody to sell the code, if he want to do that.

If you are a better programmer, then try to outperform the current version of lzturbo.

[m^2]

m^2, in this case you should leave this forum since it was developed by a commercial company and was unable without selling the licenses. and i can't beleive that you really never seen any of Hollywood films

I said I avoid products that were not created by enthusiasts. There are 2 things about that. First, enthusiasm is orthogonal with commerce. Second, I avoid, but when it conflicts with other interests, I make a pick. In this case, I chose to participate because I do like this place. If this forum moved to a google group, I would leave though.
When it comes to modern Hollywood movies, I watch c.a. 1/year; only when something has super-positive reviews. Often I'm disappointed. Though I don't watch many movies in general, maybe 1/month, maybe even less.

i don't expect that he will pay for anything, i mean that he SELLS my code. and that he make clumsy assistance for all real iranian programmers trying to sell results of their work

dnd is Iranian? Because there's no copyright for foreign works in Iran. Selling closed, GPL-derived code from abroad would be an interesting case study. IANAL, but I believe that it would be legal, though users would have to comply with GPL, as it fits in their local law.

As to whether dnd indeed used Tornado:
Like I said, I believe you. I haven't been here at the time yet, but I've read the old discussions. But I don't think I've seen a nice investigation with comparison of assembly; Some shared constants come to my mind, but they turned out to be common ones.
Frankly, the form in which you remind about the issue is not a very good one. There's an accusation and no immediately available proof. There's a lot of personal remarks. I suggest that you collect your evidence, put together in one place and link to it whenever dnd shows up.

[Bulat Ziganshin]

hamidi, how i can outperform the program i can't download?

you confirmed that you stole my code, first by using my set of entropy coders, later by saying that everyone can use Huffman coder. now you says that you can sell my GPL code - so you mean that you really do it. you have not published sources of huffman coder used in lzturbo - just pointed out to one of millions of huffman coders around. the same is with match finder - you have published variant of my own code. it's easy to check that this match finder was used only by two programs around. it will be easy to prove that i lie by publishing full code of lzturbo 0.0.1 but you will never do it. so please stop lyiong - you stealed my code and trying to sell it

[dnd]

AGAIN!!! Please show to the world where is your optimal parser. UNTIL TODAY, THERE IS NO OPTIMAL PARSER IN TORNADO!!!!

Look at the reality:
http://encode.su/threads/1722-lzturb...ll=1#post33136

and
http://encode.su/threads/1722-lzturb...ll=1#post33137

[Bulat Ziganshin]

hamidi, please show us the sources of lzturbo 001. otherwise i can't stop telling that you have stealed my entire program and sell it

optimal parser

oh, sorry, i just need to add some comments in order to help you to steal even more of my code

[dnd]

AND AGAIN!!! Please show to the world where is your optimal parser. UNTIL TODAY, THERE IS NO OPTIMAL PARSER IN TORNADO!!!!

[Bulat Ziganshin]

http://encode.su/threads/1722-lzturb...ll=1#post33136

and
http://encode.su/threads/1722-lzturb...ll=1#post33137

the former is exactly tornado matchfinder. there is no other programs using the same idea. so either you have invited the same thing as me in the same time, and even give it the same name, or ...

the later is just a huffman coder. you need to prove that it was used by the first lzturbo version with huffman, release in April 2008, otherwise it can't prove anything. are you agree with me?

[Bulat Ziganshin]

AND AGAIN!!! Please show to the world where is your optimal parser. UNTIL TODAY, THERE IS NO OPTIMAL PARSER IN TORNADO!!!!

it seems that you believe that optimal coder is hard to develop. how much time you spend developing OP for lzturbo 001? it was month or two?

[dnd]

M^2 thanks for your fairness.
Bulat seems not understanding, please ask him where to find the "optimal parser" in Tornado. UNTIL TODAY, THERE IS NO OPTIMAL PARSER IN TORNADO!!!!
You can't steal something that's not available.

[Bulat Ziganshin]

hamidi, it is why your OP is so slow? you need to steal a faster one because you can't write it yourself? may be i should write some manual to help you stealing my code? sorry, i don't know Iranian

[Piotr Tarsa]

dnd:
Can you disable the filters and redo the tests?

Also what about CPU times in addition to real times?

From what I understood, zhuff is just an afterthought (ie a LZ4 with Huffman/ FSE) and more like experimentation playground than a serious competitor. Thus, to ensure apples to apples comparison you should consider my requests.

[Bulat Ziganshin]

lz4/zhuff clearly outperforms tornado (with bytecoder and hufcoder, respectively) in the terms of speed. you can see it from hamidi tests - he compares single-threaded zhuff with 8-threaded lzturbo. by dividing lzturbo results by 5, you can find that it compresses much slower than zhuff, and decompression is faster because he compared bytecoder in lzturbo with huffman coder in zhuff

tornado easily outperfroms lz4/zhuff in the terms of compression ratio since it supports dictionaries up to 1GB. of course this gives us slower compression and slightly faster decompression

[dnd]

Well there is no big difference. These are in memory benchmarks (No disk I/O is involved) without multithreading.
The option -b0 -t1 is used for zhuff, the tests are repeated 5 times and only the best timings are reported.

Hier the tests for lzturbo without any filter and without multithreading.

                       size      ratio    C MB/s     D MB/s      
app1.tar           147863826     29.4     308.46    2180.71    lzturbo 21
app3.tar            52812595     52.8     192.08    1793.70    lzturbo 21 
app4.tar           133045756     32.6     222.41    1729.11    lzturbo 21 
audio1.tar         105295578     99.4     284.64    4924.25    lzturbo 21 
cr_txt1             46960579     47.0     150.25    1203.20    lzturbo 21 
enwik8              42537930     42.5     153.95    1166.17    lzturbo 21 
game1.tar          279854797     39.9     209.22    1668.98    lzturbo 21 
game2.tar          264914277     43.7     225.23    1975.44    lzturbo 21 
img1.tar            64275996     70.6     149.44    1802.41    lzturbo 21 
src1.tar            26714171     26.4     243.45    1482.85    lzturbo 21 
ubuntu500m         168946014     33.8     217.57    1656.76    lzturbo 21 
Total:            1333221519

Note that there are also the methods 30,31,32 and 39 in LzTurbo with much better compression.

Bulat, please go watch TV! All tests are performed WITHOUT multithreading.

[Bulat Ziganshin]

hamidi, you not mentioned in the first post until now that tests are single core. so how i can check it? so it seems that after 6 years of development you have mastered second piece of your own code. my congrats! how many years we should wait before you will stop to violate german copyright laws and will publish sources of all lzturbo versions using my code and ideas stolen from my code? this includes all the forms of HT matchfinders, stolen arithmetic, huffman and bitwise encoders. so you can sell lzturbo that has only bytecoder+optimal matchfinder and it will not break my IP. everything else in lzturbo is mine

[Piotr Tarsa]

I've done quick tests on enwik8 and it seems that dnd's results are legit and he tests on single core. However, there may be some catch - Zhuff has small sliding window size while lzturbo may use bigger, thus achieve good compression without breaking a sweat on optimal parsing and good modelling. I've tested mode 30 in lzturbo and it lost it speed advantage while providing better compression. Curious to see the sliding windows compared.