
Originally Posted by
Fu Siyuan
Well, Thanks for everyone's reply! I don't reply rapidly because I'm thinking over about your advices. In fact, everytime I would spend some time to understand about your advices.

Then you should ask what was unclear 

Originally Posted by
Fu Siyuan
Actually, I havn't. I just always thought min_match=4 (my program was) so I doubt why hash2-hash3 are necessary ,at that time.
Hash2/3 can be good again in such circumstances. Because, hash 2/3 finder can be a hash table which holds a single phrase at a time. So, this means in shortly you will have most recent match. In the another word, you have shortest distance at most time.

Originally Posted by
Fu Siyuan
Today, I made some modification on my program. I set min_match=2 with hash2-hash3 added. However, the result is very bad. There are only several hundreds of 2-length matches per MB. And the performance even got worse (I'm using precise price evaluation). Then I eliminated the 2-length match , setting min_match=3 with hash-3. I got a LITTLE (maybe less than 1%) improve on most binary file but some times a little little worse on other kinds of data (most of my own testset are TARs with mixed files).
On binary data (X86, images, audio etc), you need short matches. But, if you got worser compression on text, that means in shortly you have a improper parsing strategy. You should focus on it to eliminate this effect.

Originally Posted by
Fu Siyuan
I didn't use HashTable because I have no knowledge of HT before I read BALZ and your posts several times. I think HashTable may cost so match memory with long search cycle. Hash Chain always costs 4*(WND_SIZE+HashHead) MB.
Actually hash tables are more both memory and cache friendly. Because, you need only one memory access at a single time. Also, you can limit your each hash table entry to cache line size (=64 bytes) to become more cache friendly. As to memory consumptation, you can limit hash table size very easily. IMO, hash tables are more easier to implement. However, if you need a good compression ratio you need longer search depth as usual.

Originally Posted by
Fu Siyuan
However,recently I found long-cycle makes such MF lose sense--- it would lose is speed.
Well...It's obvious already, I think 

Originally Posted by
Fu Siyuan
I considered to study and try HT MF some times.
Note that HT MF won't help on compression ratio. It's actually a good balance between speed/ratio IMO. If you aim higher compression, you need more higher search depth on HT or a more precise match finder.

Originally Posted by
Fu Siyuan
However, I have so much want to do. I want to try to write my own archiver, data analyzer, filter (I'm considering to improve the simple RGBdelta to RAR-RGB like

). The previous work binary-tree and optimal parsing are still stranded.
I just don't know which to focus on.
All of thing are related to each other. I mean you can't decide which part should be finished first. You should make route and implement that plan.

Originally Posted by
Fu Siyuan
Now my program have the similar compression ratio and speed compared to 7z-fast mode. However, on some binary files, my program gets nearly 6% worse. The parsing now is very very 7z - optimumfast like. I'm confusing how the so much differences comes out. Anyway, it's much better than my initial csc3.
Seems you use pure order-1 context for literal coding. It's good for text, but not for binary data in most time. You should discard low bits of previous symbol. Something like:
Code:
// instead of
literalContext = previousSymbol;
// use below code
literalContext = (previousSymbol & 0xF0)>>4; // discard 4 low bits

Originally Posted by
Fu Siyuan
one more question: Random Data always problem, only 1M/s on such kind of data. I said I've wrote a detector by count each symbols freq, to distinguish if such block data is bad data. But do the all detectors or filters scan the whole data for one or two extra time? Is there a faster way?
Ideal compression should have "online" property. That means you sequentially apply all required thing in single pass. Of course, there is no any strict law about that
You can use several pass or multiple model in parallel at single pass (of course, compression itself is another pass in this case). I think, main benefit of having "online" property is streamable compression/decompression which is ideal for networking.