New lossless image compressor

[encode]

Just started designing a new lossless image compressor!

Road map:

• Simple GUI interface a la Windows Photo Viewer
• CRC-based image integrity checking
• Clipboard operations like Copy/Paste of the bitmap data - i.e. you can press the Print Screen button and paste the screen-shot to the program
• Strong and reasonable fast compression

So you you will able to freely view and convert new compressed files.

The compressor will be mainly intended for 24/32-bit BMP files.

The compression scheme will look like:

BMP ->
Color space transform (Optionally) -> Delta transform (Optionally) -> CM or BWT+CM
-> Compressed BMP

CM may have 2D contexts etc.

Any suggestions or ideas?

[PAQer]

Any suggestions or ideas?

Try to beat BMF.

[Cyan]

Well, using 2D contexts for CM is the main step here for good compression ratio.
Not sure if delta transform can bring something useful on top of that.

As for BWT, i don't see the point here. It is a 1D sorting operation, it can only break correlations, imho.

[encode]

Actually the performance of an algorithm heavily depends on image:

butterfly4.bmp - 3008x2000x24

No code has to be inserted here.

As a note, PIM 2.90 for image compression use: Color Space/RGB Transformation+LPC+PPM

[Alexander Rhatushnyak]

Actually the performance of an algorithm heavily depends on image:
butterfly4.bmp - 3008x2000x24
No code has to be inserted here.

Is this BMF -s or BMF with no -s? How well does ZPAQ do on this image?

"Any suggestions or ideas?"

Keep away from ex-JPEGs. Either use my detector for ex-JPEGs, or write your own detector.
Don't use images from Maximumcompression.com, from MOC, and 24 Kodak images.

[m^2]

Or maybe concentrate on ex-JPEGs?
While it may make sense not to mix them with regular images, they too need to be compressed, in fact quite frequently.

[encode]

Is this BMF -s or BMF with no -s? How well does ZPAQ do on this image?

"Any suggestions or ideas?"

Keep away from ex-JPEGs. Either use my detector for ex-JPEGs, or write your own detector.
Don't use images from Maximumcompression.com, from MOC, and 24 Kodak images.

It was BMF with default options - in this case it's reasonable fast and efficient.

No code has to be inserted here.

With such option BMF rocks the word but takes 17 secs on my i7-2600. Too slow for me. My goal is ~1 sec on such large file. My current image codec compresses such file to 4,508,344 bytes in 0.5 sec or less. And it's not an overtuned codec that performs well only on this file - on other files it's about the same as fast BMF.

butterfly4.bmp was converted form camera RAW directly - i.e. no ex-JPEG.

I will never test on ex-JPEGs. It's the same as convert from low bitrate MP3 to WAV and think that I have a pure PCM sound quality.

I'm testing on:
+ BMPs that directly converted from camera RAWs
+ Game screenshots that natively saved as BMPs

[Alexander Rhatushnyak]

My current image codec compresses such file to 4,508,344 bytes in 0.5 sec or less.

This sounds very interesting! When will it be released?

butterfly4.bmp was converted form camera RAW directly - i.e. no ex-JPEG.

The ratio BMPsize/BMFsize is more often between 3:1 and 4:1 on such images, while on Butterfly4 it's almost 5:1.

BTW, ZPAQ has a special configuration for BMP images. Did you choose not to test it because it's in an external file?

[encode]

The ratio BMPsize/BMFsize is more often between 3:1 and 4:1 on such images, while on Butterfly4 it's almost 5:1.

I guess it's due to a massive Bokeh around a butterfly.

BTW, ZPAQ has a special configuration for BMP images. Did you choose not to test it because it's in an external file?

Just unable to correctly use those CFG files...

This sounds very interesting! When will it be released?

Well, I've just started. I've never written specialized image coders. My GUI archivers PIMPLE and PIM have intended transformations on top of the main compression algorithms. PIM has the most advanced algorithm of all my programs - color space transformation + LPC + PPMd.
Currently I'm just tuning the CM part on various images, and testing various transformations before CM part, including various LPC schemes. So it's hard to say the exact release date. The most complex part is the codec itself - it should be properly designed and tested - and I must achieve some performance - compression speed vs compression ratio - to be on par with current codecs. Time limit already defined - just because user must able to browse compressed images instantly, nearly with no delay. GUI part was designed, file structure - that will contain signature, version number and CRC - already designed too. The BMP file will be compressed as is - bit to bit identical to the original, including the header and extra information that image may contain. I'm thinking about the format/codec name. All in all I'm about to enter new world!

[kampaster]

Greetings encode!
Concerning offers!
Don't forget to let out the console version necessarily!
Thanks!

[encode]

If convert butterfly4.bmp to PNM, FLIC will compress it to 4,414,260 bytes. And FLIC is fast!
SO, what is FLIC algorithm, in brief?

[Alexander Rhatushnyak]

Time limit already defined - just because user must able to browse compressed images instantly, nearly with no delay.

Time limit may be defined if you specify
(1) what's the acceptable delay: is it half second, one second, two seconds, or ?
(2) what's the width*height*bpp of images that must be displayed with acceptable delays,
(3) on what hardware: i.e. what are the CPU and memory speeds and L1/L2/L3 cache sizes.

About FLIC:
http://www.linkedin.com/groups/FLIC-...256.S.42748372
Currently I may not describe the algorithm, even in brief, only in extremely brief.

[encode]

Time limit may be defined if you specify
(1) what's the acceptable delay: is it half second, one second, two seconds, or ?
(2) what's the width*height*bpp of images that must be displayed with acceptable delays,
(3) on what hardware: i.e. what are the CPU and memory speeds and L1/L2/L3 cache sizes.

(1) About one second
(2) Full HD screen shots (1920x1080) and 12 to 16 megapixel images. 24 or 32 bits per pixel
(3) Modern hardware - Intel Core i7 first gen, SandyBridge, and so on. On this hardware you must able to play latest 3D games

[encode]

Can I call this new format and program BIM? Is such image format exists?

[m^2]

Used by some CAD files.
https://secure.wikimedia.org/wikiped...f_file_formats

[encode]

But no such image format, seems to be...

[caveman]

BMP ->
Color space transform (Optionally) -> Delta transform (Optionally) -> CM or BWT+CM
-> Compressed BMP

Is the color space transform a lossless RGB to YCbCr transformation?
If I am not mistaken BMP uses the (A)RGB (Alpha Red Green Blue) order, have you considered to reorder the color components on each line?
i.e. instead of having this in memory:
ARGB ARGB ARGB ARGB... ARGB
turn it into
AAAA...A RRRR...R GGGG...G BBBB...B
Context would be components of the same color (instead of whole pixels triplet), or you could try to use n-3, n-6... (or n-4, n-8...) as a context.

[encode]

Is the color space transform a lossless RGB to YCbCr transformation?

Yep, kind of.

If I am not mistaken BMP uses the (A)RGB (Alpha Red Green Blue) order, have you considered to reorder the color components on each line?
i.e. instead of having this in memory:
ARGB ARGB ARGB ARGB... ARGB
turn it into
AAAA...A RRRR...R GGGG...G BBBB...B
Context would be components of the same color (instead of whole pixels triplet), or you could try to use n-3, n-6... (or n-4, n-8...) as a context.

32-bit BMPs have RGB + alpha channel. The codec will be mainly intended for 24-bit BMPs (32-bit BMPs will be supported too). I tried all your examples and much much more! Many things that should work in a head, did not worked completely in real life.

[Piotr Tarsa]

caveman:
Or even better - use a lossless transformation from MS HD Photo: http://en.wikipedia.org/wiki/JPEG_XR...sion_algorithm to decompose RGB into three separate planes and also compress them separately.

[Matt Mahoney]

What about zpaq -mbmp_j4 -b0 ?

[Alexander Rhatushnyak]

PIM has the most advanced algorithm of all my programs - color space transformation + LPC + PPMd.

On LPCB files, PIM 2.90 stops compressing at 141% (if BMPs, ~173% if PPMs). Better than RAR and FreeArc, but not as good as UHARC. Compression took more than an hour.

[encode]

What about zpaq -mbmp_j4 -b0 ?

C:\Test>zpaq -mbmp_j4 -b0 c butterly4.zpaq-bmp butterfly4.bmp
Using model bmp_j4.cfg
zpaqopt C:\Users\Ilia\AppData\Local\Temp\zpaqtmp1476_0

C:\Test>g++ -O3 -march=native -s -DNDEBUG C:\Users\Ilia\AppData\Local\Temp\zpaqt
mp1476_0.cpp c:\zpaq\zpaq.o c:\zpaq\libzpaq.o -Ic:\zpaq -o C:\Users\Ilia\AppData
\Local\Temp\zpaqtmp1476_0.exe
'g++' is not recognized as an internal or external command,
operable program or batch file.
Creating archive butterly4.zpaq-bmp.zpaq
zpaq -h -mcolorpre r butterfly4.bmp C:\Users\Ilia\AppData\Local\Temp\zpaqtmp1476
_1.out
Using model colorpre.cfg
zpaqopt C:\Users\Ilia\AppData\Local\Temp\zpaqtmp3684_0

C:\Test>g++ -O3 -march=native -s -DNDEBUG C:\Users\Ilia\AppData\Local\Temp\zpaqt
mp3684_0.cpp c:\zpaq\zpaq.o c:\zpaq\libzpaq.o -Ic:\zpaq -o C:\Users\Ilia\AppData
\Local\Temp\zpaqtmp3684_0.exe
'g++' is not recognized as an internal or external command,
operable program or batch file.

I need g++?!

[encode]

On LPCB files, PIM 2.90 stops compressing at 141% (if BMPs, ~173% if PPMs). Better than RAR and FreeArc, but not as good as UHARC. Compression took more than an hour.

PIM recognizes TIF, TGA and BMP only.
It uses filters on top of the main compression algorithm - PPMd - i.e. with the same high-order PPM as for all other files - far from perfect for transformed data. Just writing a fully featured image codec was not the goal at that time.

BWT, can you suggest a file from your test set, that will be the most suitable for testing and tuning?

[encode]

[Matt Mahoney]

I need g++?!

For JIT optimization. If it fails then ZPAQ will still work but not as fast. You can get rid of the error message with -j0, or if you want JIT then configure zpaqopt.bat to use a different compiler according to the readme.txt in the zpaq301.zip source distribution.

[encode]

Well, tested new codec - and I'm into something. Mainly I'm comparing it to BMF2 in fast mode and FLIC, since my codec is fast!

A small comparison on old Waterloo test, all files are converted to PNM:

No code has to be inserted here.



Alex, can you provide me a few reference 24-bit image files?

[m^2]

Interesting to see so big differences going both ways.

[Piotr Tarsa]

encode:
Could you provide timings and memory consumption?

[Alexander Rhatushnyak]

A small comparison on old Waterloo test, all files are converted to PNM:

According to A.C.T. ( http://compression.ca/act/act-graphics.html ) ERI32 5.1fre packs this set better, and it was released in 2002.

Interesting to see so big differences going both ways.

There are no algorithms for 8-bit images and for artificial images in FLIC 1.5.demo.

Alex, can you provide me a few reference 24-bit image files?

There are lots of images on http://www.photographyblog.com, I don't know which of them are more suitable for your needs.

[toi007]

Are you going to use color reduction? in some aprox of lossy format?
well if its lossless dont check my off topic bmp lossy format taht failed by a lot