iz: New fast lossless RGB photo compression

[cfeck]

Hi all, I finally managed to release a first preview of "iz" (ImageZero). Here some key stats:

• should be similar in speed as "arc a -mmm:3*8+1xb"
• compression rate similar to PNG for "good" photos
• BSD license

Caveats:

• only designed for 24 bit RGB natural photos
• compression of artificial images or very noisy images is bad
• no grayscale mode
• currently writes in machine-depended byte order, so files are not portable (they lack a header for now, anyway)

Some technical details:

• 3-pixel predictor
• single context value
• static Huffman tables (1 ... 6 bits per RGB pixel to encode context delta)
• branchless design (there is only one branch for the per-pixel loop, and one branch in the bitcoder to flush bitcache to memory)

More information:

• some compression results: http://skulpture.maxiom.de/playground/list-iz.txt
• as for timing results, try yourself, if you manage to build it
• source code at: https://github.com/cfeck/imagezero

I am very interested in any kind of feedback, since I am new to this forum. Thanks, Christoph

[m^2]

Decompression speed?

[cfeck]

Decompression speed?

On my system, decompression is 2.2x faster compared to PNG (comparing process time, since the disk speed is a limiting factor here...)

[Black_Fox]

Compared to PNG, it compresses 24 bit PPM files about 20 times faster, and decompresses to PPM about 2 times faster, while the compression rate is nearly as good as PNG for natural photos.

[Black_Fox]

<Sorry, doublepost>

[Stephan Busch]

Can anyone please provide a compiled executable?

[cfeck]

Can anyone please provide a compiled executable?

This is the binary I use: http://skulpture.maxiom.de/playground/iz (Linux glibc2.0, x86-32, MMX) Anyone who creates a Linux x86-64 or Windows binary should make sure the compilation options are optimal

[cfeck]

Some updates:

First (thanks to Alex for sending the missing file), I could run the complete LPCB test.

The "score" for IZ is 1,439,922,512 bytes (=148,14% of gralic111), so it is even slightly better than regular PNG in compression rate, despite the poor results for some PIA* images. In the four camera sets, IZ files are about 5% smaller than PNG; it even reaches JPEG-LS compression rate for the Olympus_xz1 set. Full results at http://skulpture.maxiom.de/playground/list-iz.txt

LPCB timings (only measuring processor time, since IZ saturates the disk):

method:encode/decode/bitrate
IZ: 91,7 s/94,4 s/3.327 bpc
PNG: 3035,0 s/191,5 s/3.401 bpc
FLIC: 656,8 s/682,4 s/2.399 bpc

Additionally, I compiled 32/64 bit RPM packages for Fedora/openSUSE/Mandriva on openSUSE build service, you can download ImageZero from http://download.opensuse.org/repositories/home:/cfeck/

In other words, if you want to reproduce on your system, we still need someone who can make Windows binaries :-/

[Black_Fox]

Some quick'n'dirty attempt for Win compile:

imagezero-imagezero>g++ -O2 -mmmx -Winline -fomit-frame-pointer -fno-rtti -fno-exceptions main.cpp encode.cpp decode.cpp portableimage.cpp table.cpp -o iz
portableimage.cpp:3:22: fatal error: sys/mman.h: No such file or directory
compilation terminated.

after adding http://code.google.com/p/mman-win32/...e/trunk/mman.h to mingw/include/sys:

Temp\cc0BzDV8.o:portableimage.cpp:(.text+0x1fa): undefined reference to 'munmap'
Temp\cc0BzDV8.o:portableimage.cpp:(.text+0x250): undefined reference to 'mmap'
collect2: ld returned 1 exit status

Not sure what to do, had just a few minutes of time.

[cfeck]

Not sure what to do, had just a few minutes of time.

Oh, if mingw does not provide "mmap", please change both "#if 1" to "#if 0" in portableimage.cpp. On my system, using "read" instead of "mmap" was slower, though.

[Black_Fox]

That helped, thanks. Win32 x86 compile for those interested

gcc version 4.5.1 (tdm-1)

[Karhunen]

I can't seem to ge a PBM/PNM binary encoded I get "can only compress 24 bit binary PPM files" here is an attachment I triedand GBMtools
says
#>gbmhdr RGB_OR_1200x1200_082.ppm
1200x1200 24bpp 4219Kb 100% Pixmap RGB_OR_1200x1200_082.ppm
imagemagick identify says same thing, and Image Alchemy says (affter I renamed it, its a DOS program) that
Reading Portable BitMap file R1.PNM
File Name: R1.PNM
Width x Height: 1200 x 1200
Number of Colours: True Colour (24 bits)
Raw size: 4320000 Actual size: 4320017 (Compression ratio: 1:1)

[cfeck]

I can't seem to ge a PBM/PNM binary encoded I get "can only compress 24 bit binary PPM files"

Got the same error message when trying the binary with "wine" under Linux, but I blamed wine for this.

To me, it looks like mingw does not implement the UNIX file operation functions correctly, so they should be ported to Windows native calls. Since I do not have Windows, I can only hope someone else can look into the issue. Using stdio calls could be used as a last resort, but they are so slow that I avoided them.

Amazing that nobody reported this earlier, when the binary was made available...

BTW, the image file you attached is fine, it compresses to 2670480 bytes with IZ.

[Karhunen]

Thanks Chris, if anyone is successful with Borland, Mars, or VC and could post a binary for it, the semantically challenged me would be grateful

[Stephan Busch]

I am getting the same error with the win32 build - no matter if the .ppm were created using PictView, XnView nor bmp2ppm.

[willvarfar]

A good candidate might be: https://gitorious.org/imagezero/imag...ge.cpp#line130

On Linux, generally, ::read() returns all bytes in one go when accessing local files; on windows, it returns bits at a time; you have to call ::read() in a loop until it returns 0 or -1.

This is also true on Linux for reading from NFS, for example; so its the right code to have.

[cfeck]

Thanks willvarfar for the hint.

The master branch on gitorious has been updated to include some fixes that could affect file reading. Also, the error messages have been improved, so that we can better see which call fails.

Black_Fox (or anyone else), could you offer an updated binary? If I didn't mess it, it should detect mingw, and automatically switch to using read instead of mmap, so no source changes should be required.

[Black_Fox]

Compilation is without problems, can someone check correct functioning on sample data?

[cfeck]

Thanks Black_Fox.

Still does not work under "wine", so let's assume it does not work on Windows either :-/ It says "fixme:msvcrt:MSVCRT__sopen_s : pmode 0x01a4 ignored", then hangs.

[Piotr Tarsa]

I suspect there's a problem with Wine. Why do you assume that Wine is more compatible with Windows programs than Windows itself? "fixme:msvcrt" - looks for me like they haven't reimplemented something from msvcrt library.

[cfeck]

I assumed mingw cannot correctly convert all POSIX calls (fstat etc.) to the Windows API... but you are right, that error message is actually from wine, so it might work in Windows.

[Karhunen]

I assumed mingw cannot correctly convert all POSIX calls (fstat etc.) to the Windows API... but you are right, that error message is actually from wine, so it might work in Windows.

I couldn't get it to process even a 75x75x24 image, hope this screenshot of running under Windoze 7 helps
tie down the area of trouble

[cfeck]

As expected, so no Windows binary.

Anyway, the source code is not really intended to be an application, but to be later embedded as a library into image processing or database applications. I think it is safe to assume that a developer is able to add the Windows file reading, if he does not have in-memory data.

If there is interest, I can post an update when the API is ready for review.

[willvarfar]

Any thoughts on compressing 3D texture model maps e.g. DXT?

[cfeck]

The source code for IZ on gitorious has been changed to use standard C++ file I/O calls instead of POSIX calls, so it should hopefully now work with MinGW. I am still looking for someone contributing Visual Studio and/or Windows file mapping support, though.

To compile:

g++ -march=corei7 -Werror=inline -fomit-frame-pointer -fno-rtti -fno-exceptions -O3 *.cpp -o iz

Replace "corei7" with your architecture, see http://gcc.gnu.org/onlinedocs/gcc/i3...4-Options.html

[Bulat Ziganshin]

it may be helpful: http://encode.su/threads/1512-Memory-mapped-files

[Stephan Busch]

can anybody compile it for me please?

Thanks in advance

[Shelwien]

http://nishi.dreamhosters.com/u/iz_20120322.rar

[cfeck]

Thanks Shelwien, but please use the flags I specified above.

You did not compile with optimization enabled, which makes the binary useless, because the way the code is written function inlining absolutely must be enabled. Also enable at least -mmmx for 32 bit platform.

[cfeck]

it may be helpful: http://encode.su/threads/1512-Memory-mapped-files

Merci, that looks indeed helpful