Super Microsoft LZX

[comp1]

Just thought I'd share this with you guys.

Microsoft's LZX has a limited window size. It was created back in the 1990's when the memory limitations were much different from today.

Microsoft does have a delta compression that is used in their update packages. Using a 0 byte file, I tested compression.

Quick test on ENWIK9:

         SIZE       D TIME
===========================
 CABARC  250764075   6.161
MSDELTA  209195904  12.636

Someone much smarter than me perhaps can test this and benefit from it.

Attached are Windows executables for testing. Use like this:

msdeltap.exe <patch file> <in file> <out file>
msdeltaa.exe <patch file> <in file> <out file>

Have fun!

[Shelwien]

Yes, old LZX supports window size up to 2M (2^21), and msdelta supports up to 32M (see https://msdn.microsoft.com/en-us/lib...exchg.80).aspx )

Then, there's also a new "LZMS" format in .wim ( https://wimlib.net/compression.html#LZMS ), which also has a curious E8 filter extension for x64.

[Zyzzyva]

At some point I might update wimlib to support 2^25 byte windows for LZX. Currently it only supports up to 2^21 bytes, though that's already more than Microsoft's WIM software which only supports 2^15 bytes. The way Microsoft extended LZX in LZX-DELTA to support 2^25 byte windows is a little ugly because the main Huffman code ends up with 2576 symbols (256+290*8 ), which is a lot. They must not have wanted to assign more than 17 bits to any offset slot, since > 17 bits doesn't work well with the LZX bitstream.

LZMS's preprocessing step is a little more than just "E8" (call relative) filtering: it also recognizes, for example, REX-prefixed x86_64 instructions that use instruction pointer-relative addressing. But the basic idea is the same.

LZX on the other hand really just does "E8 filtering". And it works, though at one point I compared it with the E8/E9 filter used by xz/7-Zip, and the xz/7-Zip one was better.

[maadjordan]

BTW Igor (the Inventor of 7z) has done a LZX compressor example called BIX (though not same format) and it was the bases of LZMA format.

Regarding Wim.. is it possible to make a LZX compressor for CHM/CAB containers.

[Shelwien]

> I compared it with the E8/E9 filter used by xz/7-Zip, and the xz/7-Zip one was better.

Afaik that one (bcj) is not especially good either. Same with LZMS one - I got interested, but ended up writing a new one with the same idea:

http://nishi.dreamhosters.com/u/E8flt-v2.rar
http://nishi.dreamhosters.com/u/x64flt_v2.rar

33,867,264 amdocl64.dll
33,867,394 amdocl64.bcj     // 7z a -m0=bcj mcpcom.bcj mcpcom.exe
33,882,265 amdocl64.bcj2    // 7z a -m0=bcj2 mcpcom.bcj2 mcpcom.exe
33,867,264 amdocl64.MSflt   // LZMS x64/E8 filter
33,867,264 amdocl64.E8v2    // e8filter.exe c mcpcom.exe mcpcom.E8v2
33,867,264 amdocl64.E8v3    // x64flt.exe c mcpcom.exe mcpcom.E8v3
33,867,410 amdocl64.x64bcj  // 7zdll_vE\x64\7z a -tpa -m0=x64flt -m1=bcj mcpcom.x64bcj mcpcom.exe  
33,882,659 amdocl64.x64bcj2 // 7zdll_vE\x64\7z a -tpa -m0=x64flt -m1=bcj2 mcpcom.x64bcj2 mcpcom.exe

[7z a -bb3 -mx=9 -mf=off %%a.7z %%a]
 8,431,899 amdocl64.dll.7z
 7,878,007 amdocl64.MSflt.7z
 7,665,896 amdocl64.bcj.7z
 7,662,790 amdocl64.E8v2.7z
 7,584,010 amdocl64.x64bcj.7z
 7,580,512 amdocl64.E8v3.7z
 7,522,934 amdocl64.bcj2.7z
 7,472,154 amdocl64.x64bcj2.7z

Of course bcj2 results would be even better if we'd let 7z properly process its streams, instead of using it as a standalone filter.
To be specific:

 7,315,941 amdocl_1.pa // 1.bat (lzma compression)
 7,165,490 amdocl_2.pa // 2.bat (plzma)

// 1.bat
7zdll_vE\x64\7z.exe a -bb3 -mf=off ^
-m0=x64flt -m1=bcj2 -m2=deltb:c8M -m3=deltb:c8M ^
-m4=plzma4:a1:mt1:d26:lc8:pb2:lp0:fb273:mc999 ^
-m5=deltb:c8M -m6=deltb:c8M -m7=plzma4:a0:mt1:d23:lc0:pb2:lp2:fb273 ^
-m8=deltb:c8M -m9=deltb:c8M -m10=plzma4:a0:mt1:d23:lc0:pb2:lp2:fb273 ^
-m11=lzma2:mt1:d20:lc0:lp0:pb0:fb273 ^
-mb00s0:1 -mb01s0:2 -mb01s1:5 -mb01s2:8 -mb01s3:11 -mb02s0:3 -mb03s0:4 -mb05s0:6 -mb06s0:7 -mb08s0:9 -mb09s0:10 ^
amdocl_1.pa amdocl64.dll

//.bat
7zdll_vE\x64\7z.exe a -bb3 -mf=off ^
-m0=x64flt -m1=bcj2 -m2=deltb:c8M -m3=deltb:c8M ^
-m4=lzma:mt1:d26:lc8:pb2:lp0:fb273:mc999 ^
-m5=deltb:c8M -m6=deltb:c8M -m7=lzma:mt1:d23:lc0:pb2:lp2:fb273 ^
-m8=deltb:c8M -m9=deltb:c8M -m10=lzma:mt1:d23:lc0:pb2:lp2:fb273 ^
-m11=lzma2:mt1:d20:lc0:lp0:pb0:fb273 ^
-mb00s0:1 -mb01s0:2 -mb01s1:5 -mb01s2:8 -mb01s3:11 -mb02s0:3 -mb03s0:4 -mb05s0:6 -mb06s0:7 -mb08s0:9 -mb09s0:10 ^
amdocl_2.pa amdocl64.dll