Results 1 to 7 of 7

Thread: compression trace tool

  1. #1
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,267
    Thanks
    200
    Thanked 985 Times in 511 Posts

    compression trace tool

    I made a new tool, similar to what I had in DOS times -
    http://shelwien.googlepages.com/windpmi.png

    It allows to trace the compression progress without modifying
    the executable and even wrapping the executable calls
    (like with timers) is not necessary, which is convenient for
    batch tests.
    The method is simple - a replacement kernel32.dll is created,
    which redirects most of functions to a real one.
    And windows has a feature which allows to use specific local
    dlls with an executable - eg. if we create a "ccm.exe.local"
    folder and put a replacement kernel32.dll there, it would be
    used instead of original one.
    Then, ReadFile and WriteFile calls are "extended" with data size
    accumulation, and a timer thread is added, which writes the trace log.

    I think this is very useful for compressor analysis, as we can
    clearly see some behavior patterns (like in the image linked above
    DC is generally faster than YBS, but greatly slows down specifically
    on redundant data (pic) - we'd not know that just by comparing
    results on concatenated CCC.)

    So, does anybody want to use this for a benchmark site or something?
    The main problem is that some custom batch plotting tool is necessary for
    good visualization.

    Here're some example traces made while compressing enwik8
    (red = memory used, green = bytes read, blue = bytes written)
    (ppmd memory size is in kilobytes, others in bytes;
    horizontal axis is time in seconds, vertical is volume in bytes)

    http://shelwien.googlepages.com/bcm8e25.png

    http://shelwien.googlepages.com/ccm_5.png

    Here we can see that CCM spends some noticeable
    time in memory initialization without any i/o.

    http://shelwien.googlepages.com/ppmd_o12_m256_r1.png

    Here we can see that ppmd really frees its memory :)
    Last edited by Shelwien; 17th August 2009 at 04:31.

  2. #2
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Awesome dude!

  3. #3
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    wow!
    ---
    The method is simple
    - a replacement kernel32.dll is created,
    which redirects most of functions to a real one.
    And windows has a feature which allows
    to use specific local dlls with an executable
    - eg. if we create a "ccm.exe.local" folder
    and put a replacement kernel32.dll there,
    it would be used instead of original one.
    Then, ReadFile and WriteFile calls are "extended"
    with data size accumulation, and a timer thread is added, which writes the trace log.
    ---
    awesome!

    from where do you have such information?

    best regards

  4. #4
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,267
    Thanks
    200
    Thanked 985 Times in 511 Posts
    > from where do you have such information?

    http://msdn.microsoft.com/en-us/libr...00(VS.85).aspx
    http://msdn.microsoft.com/en-us/libr...75(VS.85).aspx
    http://msdn.microsoft.com/en-us/libr...86(VS.85).aspx

    Though I only tested this on XP SP2 and SP3.
    Anyway, the link to source/demo of the trace tool is available
    in the irc channel :)

  5. #5
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    According to the first doc, you can't redirect known DLLs like kernel32.dll unless you remove them from the list in the registry key HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Contro l\Session Manager\KnownDLLs
    I assume that's what you did?

  6. #6
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,267
    Thanks
    200
    Thanked 985 Times in 511 Posts
    Dunno, I didn't do anything special for that.
    Though its true that it didn't want to just run my kernel32.dll
    from the program's directory.
    But it does use it when I put it into program.local directory - both
    on my XPs and osman's Vista x64

  7. #7
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    Heh! Another Micro$oft security hole

Similar Threads

  1. Replies: 39
    Last Post: 10th April 2014, 22:26
  2. LZBW1 - compression tool by another newbye :)
    By stfox in forum Data Compression
    Replies: 4
    Last Post: 28th April 2009, 16:33

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •