BIM (a new lossless image compressor) is here!

[encode]

http://encode.narod.ru/

Please welcome the very first version of my new lossless image compressor!

Also note that this version is kind of a "core" or "bare-bone" of my new image codec - BIM can compress faster with a slightly worse compression, also BIM can compress notable better at the cost of processing speed.

Anyway, please enjoy new release!




UPDATE:
BIM v0.02 supports 24-bit BMP, TGA and PNM/PPM images and has a higher compression

[Shelwien]

CRW_3473                 CRW_3475                 CRW_3480
9843646  1.578s  1.734s  9259860  1.531s  1.687s  10806540  1.672s  1.828s // bim 01
9633421  0.297s  0.328s  8942837  0.281s  0.312s  10629911  0.297s  0.328s // qlic v1.demo
9606709  5.453s  0.938s  8810824  5.187s  0.937s  10642841  5.313s  0.969s // bcif 1.0 beta
9262732  2.344s  1.532s  8514152  2.328s  1.500s  10127448  2.437s  1.594s // bmf 2.01
9258959  1.844s  1.953s  8450343  1.813s  1.937s   9908162  1.938s  2.047s // flic v1.4.demo
9198483  1.438s  1.609s  8586752  1.406s  1.594s  10072920  1.437s  1.640s // flic v2.1.demo
8682920 37.281s 32.688s  7899272 33.234s 29.203s   9490224 36.172s 32.437s // bmf 2.01 -S 
8681236 91.593s 33.781s  7898968 84.766s 29.219s   9490224 96.031s 31.547s // bmf 2.01 -S -Q9

images are 2840x2128x24bit files converted from canon raws.

[Alexander Rhatushnyak]

1 204 847 739 bytes on LPCB images, all decompressed files were bit-exact.

Why flic v1.4.demo instead of 2.1.demo? There is no doubt the latter must be used.

[encode]

Thanks for testing! I hope you'll add it to the LPCB page too.

2Shelwien
Thanks for good examples of "bad" images for BIM. I will keep these for testing!

[Shelwien]

added flic 2.1 - 1.4 was the first link in http://www.imagecompression.info/gralic/
Looks like it was better though.

[encode]

Currently working on BIM v0.02. This new version will feature BMP image support, support for PPM images with comments, it will be slightly faster and 100% compatible with current version.
As an example, please download this packed BMP file and unpack it with the current BIM v0.01! BIM image format is extremely flexible!

[encode]

Well, new version will have a higher compression at higher speed - i.e. with same complexity + some optimizations. And it will be not compatible with the current one!

[encode]

Some testing results:

cow.ppm
bim001: 18361655->7207617
bim002: 18361655->6956958

canon24.ppm
bim001: 53747729->22567517
bim002: 53747729->21844975

DSC_8341.ppm
bim001: 36636689->15934249
bim002: 36636689->14886643

CRW_3480.ppm
bim001: 18130577->10806497
bim002: 18130577->10426843

But, in some cases:
sony24.ppm
bim001: 48098321->19324061
bim002: 48098321->19689776

The most important thing to note is that the new BIM v0.02 IS FASTER than BIM v0.01! Overall it has notable better compression (Except sony24.ppm and some weird artificial files)

[encode]

Added TGA image support. Please note that it's better to test BIM with PPM/PNM or TGA files, instead of BMPs. BMP files contain extra bytes for row alignment, latter may contain noise. Since BIM is TRULY lossless (all header info and comments are kept untouched) it will compress that noise/redundant bytes as well. With a fake PPM header, as example, you may compress and decompress any file with no loss, even a text file!

C:\bim\Release>bim c cow.ppm cow.bim
Compressing cow.ppm: 18361655->6956962 in 0.624s

C:\bim\Release>bim c cow.tga cow.bim
Compressing cow.tga: 18361656->6956572 in 0.624s

C:\bim\Release>bim c cow.bmp cow.bim
Compressing cow.bmp: 18367770->7236460 in 0.624s

[Matt Mahoney]

That doesn't look right. A normal BMP header is only 54 bytes, and end of row padding should be predictable.

[encode]

The file shown is the worst case scenario. Looks like these padding bytes add noise.

[encode]

An opposite example:

C:\bim\Release>bim c PIA12203.ppm PIA12203.bim
Compressing PIA12203.ppm: 353296335->95418720 in 10.873s

C:\bim\Release>bim c PIA12203.bmp PIA12203.bim
Compressing PIA12203.bmp: 353311650->95378096 in 10.966s

[Alexander Rhatushnyak]

C:\bim\Release>bim c cow.ppm cow.bim
Compressing cow.ppm: 18361655->6956962 in 0.624s
...
C:\bim\Release>bim c cow.bmp cow.bim
Compressing cow.bmp: 18367770->7236460 in 0.624s

in PPM files the lines are in top-to-bottom order,
in BMP they are bottom-to-top,
this may be the reason.

[encode]

BIM v0.02 has been released!

http://encode.narod.ru/

Please enjoy new release!



Just a few notes. As I said, new version is slightly faster, supports 24-bit BMP, TGA and PNM/PPM images. It also has a slightly more complex internals than BIM v0.01, but I'm aiming on speed, so I'll continue to keep it as simple as possible. As example, changing CM Counters to more advanced ones will notable improve compression at the cost of 1.5-2X slowdown - it's not the case for a fast image compressor.

[Bloax]

After long years of lurking, I couldn't really miss out to post that BIM performs pretty badly on some images.

Which could of course be explained by the primary focus being photographic images, but that's far from the only use for a lossless image format.
So yeah, uh, hi?

[encode]

BIM is designed for large (12+ MP), true-color, photographic images. It is just not intended for such artificial images.

However, about such "bad" performance. Check out how some other image compressors perform on one of your test files:

No code has to be inserted here.

Yes, for artificial images BIM v0.01 is just better. (Better pattern recognition blah blah blah)

[Bloax]

Oh noes!

*** LZ4 v1.3b, by Yann Collet (Sep  3 2012) ***
Compressed Filename is : H:\Test.ppm.lz4
Detected : 2 cores
Compressing H:\Test.ppm using 1 threads (compression level = 0)
Compression completed : 3230KB --> 317KB  (9.82%) (324960 Bytes)
Total Time : 0.02s ==> 194.6MB/s
(CPU : 0.02s = 92%)

Though this is definitely as artificial as images get, it's literally one huge scene composed from a set of little images. (Which is why it's so compressible.)

Also oh dear, BCIF sure did bad on it. :s

[encode]

[encode]

C:\bcm\x64\Release>bcm test.ppm
BCM 0.12 Experimental BWT compressor
Copyright (C) 2010 Ilia Muraviev

Compressing...
3307588 -> 54477 in 0 sec

[Bloax]

The point really is that it's kind of silly that lz4 in maximum Speed Racer mode is that much better.

[Matt Mahoney]

   29,493 test1.paq8px v69 -7
   54,461 test1-3.zpaq
   54,884 test1-8.zpaq
   54,886 test1-7.zpaq
   54,894 test1-6.zpaq
   54,903 test1-5.zpaq
   64,877 test1.dmc 1000000000
   67,690 test1-4.zpaq
   73,067 test1.bz2 -9
   75,602 test1.7z -mx
   80,261 test1.pmd -o8 -m256 -r1
  108,830 test1-2.zpaq
  125,027 test1.gz -9
  152,642 test1.ctw -d6 -n16M -f16M
  158,299 test1-1.zpaq
  419,393 test1.Z
2,407,842 test1.fpaq0
3,307,588 test.ppm

zpaq -method 3 is BWT. 4..8 are CM. 1..2 are LZ77.

[Alexander Rhatushnyak]

1 161 296 397 bytes on LPCB images, all decompressed files were bit-exact.

[encode]

Thanks for testing!

[encode]

Results with different counters - BIM v0.02 - Fast counters, BIM v0.02s - Slow counters (1.5X slower)

No code has to be inserted here.

This means only one thing - current counters are crap. Will search for a faster version of these "slow" counters.

[Alexander Rhatushnyak]

This means only one thing - current counters are crap.

Not necessarily. Look at them from the other side: when you switch from slow counters to fast, compression quality is ~1% worse, but speed is 1.5 times higher!
Actually, even less than 1% on average most likely,
e.g. PIA12203.ppm: 95418720/94794028 = 1.00659

[encode]

Yep, in terms of efficiency - with no doubt. But my current goal with BIM - a higher compression! BIM must beat RKIM and a couple of others. At the same time, I have no room for an additional complexity that will slow it down, easily. FLIC defined a bargain.
Like I said, I have a bunch of tricks that will significantly improve BIM's compression performance - at the cost of 2X-10X speed penalty - no reason to do that since we have PAQ8(im) and such. So currently I'm trying to squeeze-in some of the ideas, in simplified and optimized forms.
Like these slow counters might be simulated with much faster FSM-based ones. Just currently, I had no luck - FSM building and optimization is not the easiest task.

[encode]

GARTEST - re-rendered with a newer ViewNX software + NEF(RAW)
http://compressme.net/gartest.rar

Currently, I'm working on BIM v0.03

Some results on DSC_8341.ppm:
BIM v0.02 -> 14,871,253 bytes
BIM v0.03 -> 14,363,389 bytes

[kampaster]

Hi Ilya.
No code has to be inserted here.

[Matt Mahoney]

I used ImageMagick convert to convert .tga to .bmp and .pmm. Tests with zpaq were on the .tga file.

   544,707 AIMP_free-m1.zpaq
   494,449 AIMP_free-m2.zpaq
   518,582 AIMP_free-m3.zpaq
   355,052 AIMP_free-m4.zpaq
   308,943 AIMP_free-m5.zpaq
   279,088 AIMP_free-m6.zpaq
16,439,334 AIMP_free.bmp
 2,050,086 AIMP_free.bmp.bim
16,437,860 AIMP_free.ppm
 2,064,893 AIMP_free.ppm.bim
16,437,887 AIMP_free.tga
 2,049,953 AIMP_free.tga.bim
   731,038 AIMP_free.zip

[encode]

Once again - BIM was not designed for such images. BIM is for true color pictures from your camera...