nosso hack

[Shelwien]

As some may remember, there was this thing - http://encode.su/threads/140-Nosso-C...ion-technology

And recently I stumbled on it again, and really looked into it this time.

https://sites.google.com/site/shelwi...edirects=0&d=1

This is a hacked version with removed encryption and archive size restrictions.

And results are as follows:

1. Nosso actually uses lzma for compression, although encrypts the code to avoid detection.
And its even plain lzma with default parameters - they didn't even bother to optimize these.

2. There's still a good thing in it - there's an .exe preprocessor with likely the best known results.

806,912 oodle213.dll  // sample 32-bit executable 
267,433 oodle213.7z   // 7z ultra (bcj2)
246,457 00000000.lzma // nosso payload (see test.bat)

199,104 oodle213.dll.paq8px // paq8px69 -7 
196,922 00000000.unp.paq8px // paq8px69 -7

However it only supports _valid_ 32-bit PE/COFF files (ie no dumped/unpacked exes).
And its not just an easy to hack off check - there's a PE header parser and everything.

Thanks to kampaster for providing the original executable.

[Shelwien]

Ok, here's more

https://sites.google.com/site/shelwi...edirects=0&d=1

This is a durilca 0.5 patched with equivalent of old -l option, to be specific
dur_flt.exe creates durilca.dmp file with all the preprocessed data, and
dur_seg.exe creates a separate file for each preprocessor segment.
Results:

1,005,425 durilca-t3.dmp        // dur_flt.exe e -o2 -t3 oodle213.dll
1,002,941 durilca-t1.dmp        // dur_flt.exe e -o2 -t1 oodle213.dll
  807,678 dispack.bin           // dispack.exe e oodle213.dll dispack.bin
  198,025 dispack.bin.paq8px    // paq8px_v69_sse2.exe -7 dispack.bin
  193,266 durilca-t3.dmp.paq8px // paq8px_v69_sse2.exe -7 durilca-t3.dmp
  192,001 durilca-t1.dmp.paq8px // paq8px_v69_sse2.exe -7 durilca-t1.dmp

So I guess nosso filter isn't really the best in the end.

[mpais]

Thank you, some really nice results. I decided to run some tests with EMMA to see if it could benefit from such a preprocessor.

File: oodle213.dll, 806.912 bytes

198.037 bytes, paq8px_v75 -8
197.672 bytes, paq8pxd_v16 -s15
186.772 bytes, EMMA 0.1.10 x86, Preset "Best"
204.356 bytes, EMMA 0.1.10 x86, Preset "Best", x86/x64 model disabled

File: durilca-t1.dmp, 1.002.941 bytes

191.551 bytes, paq8px_v75 -8
191.622 bytes, paq8pxd_v16 -s15
193.253 bytes, EMMA 0.1.10 x86, Preset "Best"
195.023 bytes, EMMA 0.1.10 x86, Preset "Best", x86/x64 model disabled

File: 00000000.unp, 800.811 bytes

196.584 bytes, paq8px_v75 -8
197.287 bytes, paq8pxd_v16 -s15
199.944 bytes, EMMA 0.1.10 x86, Preset "Best"
200.451 bytes, EMMA 0.1.10 x86, Preset "Best", x86/x64 model disabled

So, when going for maximum compression, both preprocessors actually hurt compression, since EMMA already has a model for x86/x64 code.
But disable that model and Durilca's preprocessor gets a really good improvement. So it may be interesting to have some sort of fast
preprocessor, currently EMMA's x86/x64 transform only does relative-to-absolute address conversions. I need to do more tests.

[Shelwien]

Check also
http://www.farbrausch.de/~fg/code/disfilter/
https://sites.google.com/site/shelwi...edirects=0&d=1 (this one is a dummy version of durilca filter)
https://github.com/ChromiumWebApps/c...ster/courgette (-dis option)

[Bulat Ziganshin]

dispack is also incorporated into freearc/fazip:

M:\>fazip64.exe dispack070 z:\100m nul
100%: 100,000,000 -> 98,276,089: 98.28%
Cpu 142 mb/s (0.671 sec), real 135 mb/s (0.708 sec) = 95%

[Shelwien]

So https://github.com/ChromiumWebApps/c...ster/courgette

Courgette is an exe diff utility used for chrome browser updates.
In particular, it includes a disasm preprocessor.

http://nishi.dreamhosters.com/u/courgette2016.rar (exe only)
http://nishi.dreamhosters.com/u/courgette2016_v0.rar (exe+sources)

Must have exactly one of:
  -supported -asm, -dis, -disadj, -gen or -apply, -genbsdiff or -applybsdiff.
Usage:
  courgette -supported <executable_file>
  courgette -dis <executable_file> <binary_assembly_file>
  courgette -asm <binary_assembly_file> <executable_file>
  courgette -disadj <executable_file> <reference> <binary_assembly_file>
  courgette -gen <v1> <v2> <patch>
  courgette -apply <v1> <patch> <v2>

It supports pe/elf x86/x64/arm which is kinda cool.
But results are a bit uncertain...

Effects of disasm filter on file compression:

1,606,656 7z_dll 1,642,623 7z_dll_dis // courgette.exe -dis 7z_dll 7z_dll_dis (x64)
  537,691 1.plz    508,689 2.plz      // plzma c
  549,773 1.7z     521,212 2.7z       // 7z a -mx=9 -m0=lzma     
  497,590 1a.7z    521,212 2a.7z      // 7z a -mx=9 -myx=9 -m0=lzma     
  360,854 1.paq8p  384,534 1.paq8p    // paq8p -7

As to actual diff performance, I guess it improved (see also http://encode.su/threads/582-Executa...ration-methods )

Performance comparison of actual diff methods (bsdiff/bsdiff_sh/courgette-with-disasm)

2,031,104 7z1.dll
2,046,976 7z2.dll
2,079,913 7z1_dis       // courgette.exe -dis 7z1.dll 7z1_dis
2,096,071 7z2_dis       // courgette.exe -dis 7z2.dll 7z2_dis
  365,079 12_dll_bs     // courgette.exe -genbsdiff 7z1.dll 7z2.dll 12_dll_bs
  353,132 12_dll_bsh    // bsdiff_sh2 7z1_dis 7z2_dis 12_dis_bsh  
  231,263 12_dis_bs     // courgette.exe -genbsdiff 7z1_dis 7z2_dis 12_dis
  231,239 12_dis_bsh    // bsdiff_sh2 7z1.dll 7z2.dll 12_dll_bsh  
  116,738 12_dll        // courgette.exe -gen 7z1_dis 7z2_dis 12 (implies -dis?)
  159,811 12_dll_bs.7z  // 7z a -mx=9 -myx=9 12_dll_bs 12_dll_bs
  150,584 12_dll_bsh.7z // 7z a -mx=9 -myx=9 12_dis_bsh 12_dis_bsh
  121,732 12_dis_bsh.7z // 7z a -mx=9 -myx=9 12_dll_bsh 12_dll_bsh
  120,376 12_dis_bs.7z  // 7z a -mx=9 -myx=9 12_dis_bs 12_dis_bs
   75,730 12_dll.7z     // 7z a -mx=9 -myx=9 12_dll 12_dll

[schnaader]

Ah, thanks for posting, always wanted to try this out, but never found time for it.

Might be useful for ooffice from silesia. Similar to your results, it only works for compressors without x86 transform. Also tried with the .so files from mozilla, but these are some unsupported outdated elf format.

 6,152,192    ooffice
 6,429,813    courgette -dis

 2,860,954    Precomp v0.4.5 (similar to plain bzip2)
 2,509,785    courgette -dis | Precomp 0.4.5

 2,427,223    7-Zip Ultra LZMA2
 2,135,358    courgette -dis | 7-Zip Ultra LZMA2

 1,766,557    zpaq v7.05 -method 7
 1,856,912    courgette -dis | zpaq v7.05 -method 7

 1,440,341    paq8l -8
 1,576,733    courgette -dis | paq8l -8

A detailed description of the process can be found here. I wondered if the relative->absolute transform could be combined with courgette, but it seems that it already does something similar:

[...]

• The Disassembler builds a list of addresses referenced by the machine code, numbering each one.
• The Disassembler replaces and address used in machine instructions with its index number.

[...]

[Shelwien]

> replaces any address used in machine instructions with its index number

Its a diff thing actually, and isn't very useful for exe preprocessing.
As bcj2/flt32 shows, it would be better to move addresses to a different stream instead,
then same code patterns with different addrs would match.

But I think this is the first x64 disasm filter ever, so its still great :)