Yepp, I got it. Circular hash chains rules!

My target was made LZPM (16 MB dictionary) compressor which will use less than 100 MB at compression while keeping good match search (near to brute force - i.e. finding all matches, note it is possible with ROLZ).

And I did it! New circular hash chains works!

Check out some results:

PAK0.PAK (183,997,730 bytes):
LZPM 0.03: 90,866,467 bytes; memuse: 176 MB
LZPM 0.04v1: 90,896,664 bytes; memuse: 80 MB
LZPM 0.04v2: 91,012,829 bytes; memuse: 48 MB
LZPM 0.04v3: 91,161,996 bytes; memuse: 32 MB (2*N [!])

ENWIK8 (100,000,000 bytes):
LZPM 0.03: 29,248,641 bytes; memuse: 176 MB
LZPM 0.04v1: 29,297,905 bytes; memuse: 80 MB
LZPM 0.04v2: 29,406,000 bytes; memuse: 48 MB
LZPM 0.04v3: 29,613,694 bytes; memuse: 32 MB (2*N [!])

TraktorDJStudio3.exe (29,124,024 bytes):
LZPM 0.03: 5,927,489 bytes; memuse: 176 MB
LZPM 0.04v1: 5,927,529 bytes; memuse: 80 MB
LZPM 0.04v2: 5,933,261 bytes; memuse: 48 MB
LZPM 0.04v3: 5,953,096 bytes; memuse: 32 MB (2*N [!])

gimp-2.0.0.tar (78,745,600 bytes):
LZPM 0.03: 10,736,681 bytes; memuse: 176 MB
LZPM 0.04v1: 10,773,046 bytes; memuse: 80 MB
LZPM 0.04v2: 10,847,118 bytes; memuse: 48 MB
LZPM 0.04v3: 10,970,042 bytes; memuse: 32 MB (2*N [!])

80 MB version is pretty cool... and it's faster!

Continue testing...