4x4 bug?

[m^2]

I found a strange thing happening - when I compress the same files multiple times, I got different output archives.
It happens in modes 4..7, also with single threaded compression. They all decompress correctly.
Is it a bug or just unusual, but expected behaviour?
It might be related to this, that's the same test set, just tared.
Standalone Tornado doesn't seem to have this issue.

BTW, encode, could you drop or reduce search length word limit?
4 characters is 1 too much.

[Simon Berger]

You find something like this in most lz based compressors. It's absolut normal. I or someone else could give you a technical answer if you want to.

[m^2]

Could you do this, please?
I didn't mean that different modes work differently, but that each of them is non deterministic.

[Bulat Ziganshin]

i'm also intrigued )

[Simon Berger]

Yes I understood what you mean. I'm not that into how most of the programs work but I am on the way of programming my own lz based compressor at the moment. I write my output into a unititialized buffer and at writing the buffer to the file there are most times some bits not used (at the last byte of the buffer) which could have any values but it does not matter because they won?t be interpreted at any time.
I hope what I wrote is understandable

[toffer]

I always thought the main reason is a uninitialized hash table (for LZ77 based stuff)?!

[m^2]

Yes I understood what you mean. I'm not that into how most of the programs work but I am on the way of programming my own lz based compressor at the moment. I write my output into a unititialized buffer and at writing the buffer to the file there are most times some bits not used (at the last byte of the buffer) which could have any values but it does not matter because they won?t be interpreted at any time.
I hope what I wrote is understandable

But it shouldn't change the output size?
There's like 0.1% variation with 2 threads. I run total of 15 tests ( 3*3 + 1*6 ) in this mode and only 1 result was repeated.
With 1 thread it was 12 tests (4*3) and only 1 result was 1 byte off, so this is probably related to multithreading.

And Bulat..grab the files and see what's the reason yourself.

[Bulat Ziganshin]

i've found bug which should lead to this problem. at the time when i wrote 4x4 i comletely firget that tornado has 1 global variable left- - it's used to detect binary tables in data (like delta filter does). i don't yet tested fixed 4x4 but hope that now it will compress binary data 1-2% better and show stable results. thank you for reporting the problem!

[m^2]

i've found bug which should lead to this problem. at the time when i wrote 4x4 i comletely firget that tornado has 1 global variable left- - it's used to detect binary tables in data (like delta filter does). i don't yet tested fixed 4x4 but hope that now it will compress binary data 1-2% better and show stable results. thank you for reporting the problem!

How about the single threaded mode?
There was 1 result 1 byte different too. I can't find the logs, strange...possibly I lost them. It may be my mistake, but such assumptions are dangerous.
I'll try to recreate the problem.

[Bulat Ziganshin]

well, i've already fixed bug with unitialized hash (exactly as Simon said, although it was unintentional) so we may expect that there were 2 bugs - one small and one worth 1% of compression. i've attached fixed 4x4, can you please test it thoroughly?

[m^2]

I did ~100 tests and everything seems OK.

[Bulat Ziganshin]

thanks!