MM type detector

[Bulat Ziganshin]

Christian said that simple entropy meter don't helped him to properly detect MM data and grzip's MM detection algorithm, as i found, is also more or less broken

so i developed MM type detector which compares entropy (measured by results of order-0 arithmetic encoding) of various encodings. in my tests it properly found data type of all the greyscale, rgb, wave mono and stereo files i've tried. moreover, it even properly detects 16/32-bit tables what don't need substracting

http://www.haskell.org/bz/mmdet.rar

it's great companion for http://www.haskell.org/bz/tta.rar

[Christian]

Christian said that simple entropy meter dont helped him to properly detect MM data

I think I worded this in a wrong way. For data detection only, o0 is fine - in almost all cases.

The problem I had was this:

...As a rule of thumb i recommend you to use any MM compression if it gives result at least 5% better than 8bit model...

For WAV data, integer tables and the like this works nice.
But for RGB filtering, o0 sometimes indicates that a filter gives a boost in compression by 20+% - but actually the filter hurts compression. I found many images with this pathological property. A stronger compression test almost solved this problem (I ended up using an extremly fast algorithm which mimics LZ + Ari).
So, to sum it up. I had problems with the combination of a o0 compression test and RGB filtering. But it would be really cool, if youd find a way around this!

Best regards,

Christian

[Bulat Ziganshin]

just try my program. it compares compressed size with results of order-0 model (i.e. it applies order-0 model both to original data and data after preprocessing, with a separate encoding for each channel).

moreover, it actually uses combined model - order-0 for values in -128..127 range, and logarithmic encoding for data outside of this range (of course, this used only to shorts/24bits/longs)

if this program fails i want to see these files (email?). also, there are some "to do's" in the program, may be someone will contribute code that fixes them

[Christian]

if this program fails i want to see these files

Theres still a misunderstanding here. The data detection is not the problem and does not fail. The problem is the decision if the data filter should be applied. I know this sound confusing, because you assume, that if RGB is detected a RGB filter should improve compression. But sadly, its not the case.

Just take rafale.bmp from the SFC test. The detection says "8bit * 3 channels, diffed" - which obviously seems right, because rafale.bmp IS a picture.
But have you ever tried to actually apply "3-byte-diff" filter to rafale.bmp? It badly hurts compression. If youre using more advanced RGB filters this problem even gets worse on some files.

e.g. WinRAR:
best - ~803K
forced truecolor filter - 1407K
best with disabled truecolor filter, but "3-byte-diff" - 997K

e.g. 7-Zip:
PPM o4 - 747K
PPM o4 with "3-byte-diff" - 978K
LZMA ultra - 954K
LZMA ultra with "3-byte-diff" - 1094K

Remember, that MM detection adviced us to use "3-byte-diff" for rafale.bmp.

BEST model is 8bit * 3 channels, diffed (45.3% compared to 8bit)

[Uwe Herklotz]

The data detection is not the problem and does not fail. The problem is the decision if the data filter should be applied. I know this sound confusing, because you assume, that if RGB is detected a RGB filter should improve compression. But sadly, its not the case.

Exactly!

There is one "easy" way to get around the "decision"-problem. If a certain mm type is detected then run models for original and filtered data in parallel and decide on-the-fly which to use for encoding, i.e. based on recent success of original/filtered models the next symbol is coded with the better one.

[Bulat Ziganshin]

thanks, now i understood that you mean. btw, i even tried to apply tta (forced to the 3channels*8bit mode) to bmp files. the only exception whose compression was worse than using grzip, was rafale. i concluded that tta, being wave filter, can't provide guaranteed compression for bmp files

now it seems that the problem is only detection code itself. if it will be improved, i can use tta both for bmp and wave files

[Bulat Ziganshin]

btw. is there any other files with such good behaviour or radale is th only one?

i've optimized mmdet and simplified its usage in 3rd-party code. now it is included as a part of http://www.haskell.org/bz/tta.rar - see mmdet.cpp

[Christian]

btw. is there any other files with such good behaviour or radale is th only one?

There are many files which show such a behaviour. e.g. CCM on the squeeze-chart image:

1.17b: 53.891.837
1.17e: 46.951.192

1.17e introduced improvements to the detection/decision (LZA) - the filter itself wasnt changed. Here, the MM filter hurts compression only on parts of the image.

Some other examples from the waterloo set with RAR:

clegg:
forced MM: 1146K
best: 503K

frymire:
forced MM: 986K
best: 204K

serrano:
forced MM: 382K
best: 89K

Youll find many images with this property. Just do some tests.

[Uwe Herklotz]

btw. is there any other files with such good behaviour or radale is th only one?

Generally, MM filters for images work best with photo-images, i.e. [raw/uncompressed true-color] pictures taken with a digicam or similar ones. This relates to simple delta-filters but is true for most complex 2D-filters as well.

There are many true-color (24/32 bit) image types which may mislead MM detection and therefore filter application hurts compression. This may include:
+ hand-drawn images (e.g. cartoons, pics for presentation slides)
+ computer-generated images (e.g. desk screenshots, simple raytracing pictures)
+ images stored as true-color but using reduced color space (e.g. rafale.bmp - IIRC)
+ photo-images with artefacts (e.g. ex-lossy compressed, scaled with simple algorithms)
+ photo-image with certain filter levels applied (e.g. try extensive blur or softening)
+ tbc....

Note: its possible to create similar "bad" examples for audio data as well, but for audio you wont see such things in practice, whereas such special images can be seen quite often.

[Stephan Busch]

Dear Bulat,

where can I get the latest FreeArc for win version? I want to let it run over my testsets, preferably with TTA implementation.
The last version I saw was without TTA audio compressor.

[Bulat Ziganshin]

it is not yet released, i work on it. you can download TTA compressor itself as http://www.haskell.org/bz/tta.rar

last published freearc version is availlable as http://www.haskell.org/bz/FreeArc-win32.7z

(you should put arc.ini and arc.groups in the same directory as arc.exe)