Results 1 to 11 of 11

Thread: MM type detector

  1. #1
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    Christian said that simple entropy meter don't helped him to properly detect MM data and grzip's MM detection algorithm, as i found, is also more or less broken

    so i developed MM type detector which compares entropy (measured by results of order-0 arithmetic encoding) of various encodings. in my tests it properly found data type of all the greyscale, rgb, wave mono and stereo files i've tried. moreover, it even properly detects 16/32-bit tables what don't need substracting

    http://www.haskell.org/bz/mmdet.rar

    it's great companion for http://www.haskell.org/bz/tta.rar

  2. #2
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 153 Times in 18 Posts
    Quote Originally Posted by Bulat Ziganshin
    Christian said that simple entropy meter dont helped him to properly detect MM data
    I think I worded this in a wrong way. For data detection only, o0 is fine - in almost all cases.

    The problem I had was this:
    Quote Originally Posted by Source
    ...As a rule of thumb i recommend you to use any MM compression if it gives result at least 5% better than 8bit model...
    For WAV data, integer tables and the like this works nice.
    But for RGB filtering, o0 sometimes indicates that a filter gives a boost in compression by 20+% - but actually the filter hurts compression. I found many images with this pathological property. A stronger compression test almost solved this problem (I ended up using an extremly fast algorithm which mimics LZ + Ari).
    So, to sum it up. I had problems with the combination of a o0 compression test and RGB filtering. But it would be really cool, if youd find a way around this!

    Best regards,

    Christian

  3. #3
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    just try my program. it compares compressed size with results of order-0 model (i.e. it applies order-0 model both to original data and data after preprocessing, with a separate encoding for each channel).

    moreover, it actually uses combined model - order-0 for values in -128..127 range, and logarithmic encoding for data outside of this range (of course, this used only to shorts/24bits/longs)

    if this program fails i want to see these files (email?). also, there are some "to do's" in the program, may be someone will contribute code that fixes them

  4. #4
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 153 Times in 18 Posts
    Quote Originally Posted by Bulat Ziganshin
    if this program fails i want to see these files
    Theres still a misunderstanding here. The data detection is not the problem and does not fail. The problem is the decision if the data filter should be applied. I know this sound confusing, because you assume, that if RGB is detected a RGB filter should improve compression. But sadly, its not the case.

    Just take rafale.bmp from the SFC test. The detection says "8bit * 3 channels, diffed" - which obviously seems right, because rafale.bmp IS a picture.
    But have you ever tried to actually apply "3-byte-diff" filter to rafale.bmp? It badly hurts compression. If youre using more advanced RGB filters this problem even gets worse on some files.

    e.g. WinRAR:
    best - ~803K
    forced truecolor filter - 1407K
    best with disabled truecolor filter, but "3-byte-diff" - 997K

    e.g. 7-Zip:
    PPM o4 - 747K
    PPM o4 with "3-byte-diff" - 978K
    LZMA ultra - 954K
    LZMA ultra with "3-byte-diff" - 1094K

    Remember, that MM detection adviced us to use "3-byte-diff" for rafale.bmp.
    Quote Originally Posted by MMDET
    BEST model is 8bit * 3 channels, diffed (45.3% compared to 8bit)

  5. #5
    Member
    Join Date
    Mar 2007
    Posts
    34
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Christian
    The data detection is not the problem and does not fail. The problem is the decision if the data filter should be applied. I know this sound confusing, because you assume, that if RGB is detected a RGB filter should improve compression. But sadly, its not the case.
    Exactly!

    There is one "easy" way to get around the "decision"-problem. If a certain mm type is detected then run models for original and filtered data in parallel and decide on-the-fly which to use for encoding, i.e. based on recent success of original/filtered models the next symbol is coded with the better one.

  6. #6
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    thanks, now i understood that you mean. btw, i even tried to apply tta (forced to the 3channels*8bit mode) to bmp files. the only exception whose compression was worse than using grzip, was rafale. i concluded that tta, being wave filter, can't provide guaranteed compression for bmp files

    now it seems that the problem is only detection code itself. if it will be improved, i can use tta both for bmp and wave files

  7. #7
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    btw. is there any other files with such good behaviour or radale is th only one?

    i've optimized mmdet and simplified its usage in 3rd-party code. now it is included as a part of http://www.haskell.org/bz/tta.rar - see mmdet.cpp

  8. #8
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 153 Times in 18 Posts
    Quote Originally Posted by Bulat Ziganshin
    btw. is there any other files with such good behaviour or radale is th only one?
    There are many files which show such a behaviour. e.g. CCM on the squeeze-chart image:

    1.17b: 53.891.837
    1.17e: 46.951.192

    1.17e introduced improvements to the detection/decision (LZA) - the filter itself wasnt changed. Here, the MM filter hurts compression only on parts of the image.

    Some other examples from the waterloo set with RAR:

    clegg:
    forced MM: 1146K
    best: 503K

    frymire:
    forced MM: 986K
    best: 204K

    serrano:
    forced MM: 382K
    best: 89K

    Youll find many images with this property. Just do some tests.

  9. #9
    Member
    Join Date
    Mar 2007
    Posts
    34
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Bulat Ziganshin
    btw. is there any other files with such good behaviour or radale is th only one?
    Generally, MM filters for images work best with photo-images, i.e. [raw/uncompressed true-color] pictures taken with a digicam or similar ones. This relates to simple delta-filters but is true for most complex 2D-filters as well.

    There are many true-color (24/32 bit) image types which may mislead MM detection and therefore filter application hurts compression. This may include:
    + hand-drawn images (e.g. cartoons, pics for presentation slides)
    + computer-generated images (e.g. desk screenshots, simple raytracing pictures)
    + images stored as true-color but using reduced color space (e.g. rafale.bmp - IIRC)
    + photo-images with artefacts (e.g. ex-lossy compressed, scaled with simple algorithms)
    + photo-image with certain filter levels applied (e.g. try extensive blur or softening)
    + tbc....

    Note: its possible to create similar "bad" examples for audio data as well, but for audio you wont see such things in practice, whereas such special images can be seen quite often.

  10. #10
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    474
    Thanked 175 Times in 85 Posts
    Dear Bulat,

    where can I get the latest FreeArc for win version? I want to let it run over my testsets, preferably with TTA implementation.
    The last version I saw was without TTA audio compressor.

  11. #11
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    it is not yet released, i work on it. you can download TTA compressor itself as http://www.haskell.org/bz/tta.rar

    last published freearc version is availlable as http://www.haskell.org/bz/FreeArc-win32.7z

    (you should put arc.ini and arc.groups in the same directory as arc.exe)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •