compression trace tool

[Shelwien]

I made a new tool, similar to what I had in DOS times -
http://shelwien.googlepages.com/windpmi.png

It allows to trace the compression progress without modifying
the executable and even wrapping the executable calls
(like with timers) is not necessary, which is convenient for
batch tests.
The method is simple - a replacement kernel32.dll is created,
which redirects most of functions to a real one.
And windows has a feature which allows to use specific local
dlls with an executable - eg. if we create a "ccm.exe.local"
folder and put a replacement kernel32.dll there, it would be
used instead of original one.
Then, ReadFile and WriteFile calls are "extended" with data size
accumulation, and a timer thread is added, which writes the trace log.

I think this is very useful for compressor analysis, as we can
clearly see some behavior patterns (like in the image linked above
DC is generally faster than YBS, but greatly slows down specifically
on redundant data (pic) - we'd not know that just by comparing
results on concatenated CCC.)

So, does anybody want to use this for a benchmark site or something?
The main problem is that some custom batch plotting tool is necessary for
good visualization.

Here're some example traces made while compressing enwik8
(red = memory used, green = bytes read, blue = bytes written)
(ppmd memory size is in kilobytes, others in bytes;
horizontal axis is time in seconds, vertical is volume in bytes)

http://shelwien.googlepages.com/bcm8e25.png

http://shelwien.googlepages.com/ccm_5.png

Here we can see that CCM spends some noticeable
time in memory initialization without any i/o.

http://shelwien.googlepages.com/ppmd_o12_m256_r1.png

Here we can see that ppmd really frees its memory :)

[encode]

Awesome dude!

[joerg]

wow!
---
The method is simple
- a replacement kernel32.dll is created,
which redirects most of functions to a real one.
And windows has a feature which allows
to use specific local dlls with an executable
- eg. if we create a "ccm.exe.local" folder
and put a replacement kernel32.dll there,
it would be used instead of original one.
Then, ReadFile and WriteFile calls are "extended"
with data size accumulation, and a timer thread is added, which writes the trace log.
---
awesome!

from where do you have such information?

best regards

[Shelwien]

> from where do you have such information?

http://msdn.microsoft.com/en-us/libr...00(VS.85).aspx
http://msdn.microsoft.com/en-us/libr...75(VS.85).aspx
http://msdn.microsoft.com/en-us/libr...86(VS.85).aspx

Though I only tested this on XP SP2 and SP3.
Anyway, the link to source/demo of the trace tool is available
in the irc channel :)

[Matt Mahoney]

According to the first doc, you can't redirect known DLLs like kernel32.dll unless you remove them from the list in the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Contro l\Session Manager\KnownDLLs
I assume that's what you did?

[Shelwien]

Dunno, I didn't do anything special for that.
Though its true that it didn't want to just run my kernel32.dll
from the program's directory.
But it does use it when I put it into program.local directory - both
on my XPs and osman's Vista x64

[Matt Mahoney]

Heh! Another Micro$oft security hole