Page 1 of 4 123 ... LastLast
Results 1 to 30 of 106

Thread: New lossless image compressor

  1. #1
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts

    Cool New lossless image compressor

    Just started designing a new lossless image compressor!

    Road map:
    • Simple GUI interface a la Windows Photo Viewer
    • CRC-based image integrity checking
    • Clipboard operations like Copy/Paste of the bitmap data - i.e. you can press the Print Screen button and paste the screen-shot to the program
    • Strong and reasonable fast compression


    So you you will able to freely view and convert new compressed files.

    The compressor will be mainly intended for 24/32-bit BMP files.

    The compression scheme will look like:

    BMP ->
    Color space transform (Optionally) -> Delta transform (Optionally) -> CM or BWT+CM
    -> Compressed BMP

    CM may have 2D contexts etc.

    Any suggestions or ideas?

  2. #2
    Member PAQer's Avatar
    Join Date
    Jan 2010
    Location
    Russia
    Posts
    22
    Thanks
    3
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by encode View Post

    Any suggestions or ideas?
    Try to beat BMF.

  3. #3
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    863
    Thanks
    459
    Thanked 257 Times in 105 Posts
    Well, using 2D contexts for CM is the main step here for good compression ratio.
    Not sure if delta transform can bring something useful on top of that.

    As for BWT, i don't see the point here. It is a 1D sorting operation, it can only break correlations, imho.

  4. #4
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Actually the performance of an algorithm heavily depends on image:

    butterfly4.bmp - 3008x2000x24

    No code has to be inserted here.

    As a note, PIM 2.90 for image compression use: Color Space/RGB Transformation+LPC+PPM

  5. #5
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    237
    Thanks
    39
    Thanked 92 Times in 48 Posts
    Quote Originally Posted by encode View Post
    Actually the performance of an algorithm heavily depends on image:
    butterfly4.bmp - 3008x2000x24
    No code has to be inserted here.
    Is this BMF -s or BMF with no -s? How well does ZPAQ do on this image?

    "Any suggestions or ideas?"

    Keep away from ex-JPEGs. Either use my detector for ex-JPEGs, or write your own detector.
    Don't use images from Maximumcompression.com, from MOC, and 24 Kodak images.

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

  6. #6
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Or maybe concentrate on ex-JPEGs?
    While it may make sense not to mix them with regular images, they too need to be compressed, in fact quite frequently.

  7. #7
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Alexander Rhatushnyak View Post
    Is this BMF -s or BMF with no -s? How well does ZPAQ do on this image?

    "Any suggestions or ideas?"

    Keep away from ex-JPEGs. Either use my detector for ex-JPEGs, or write your own detector.
    Don't use images from Maximumcompression.com, from MOC, and 24 Kodak images.
    It was BMF with default options - in this case it's reasonable fast and efficient.

    No code has to be inserted here.

    With such option BMF rocks the word but takes 17 secs on my i7-2600. Too slow for me. My goal is ~1 sec on such large file. My current image codec compresses such file to 4,508,344 bytes in 0.5 sec or less. And it's not an overtuned codec that performs well only on this file - on other files it's about the same as fast BMF.

    butterfly4.bmp was converted form camera RAW directly - i.e. no ex-JPEG.

    I will never test on ex-JPEGs. It's the same as convert from low bitrate MP3 to WAV and think that I have a pure PCM sound quality.

    I'm testing on:
    + BMPs that directly converted from camera RAWs
    + Game screenshots that natively saved as BMPs

  8. #8
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    237
    Thanks
    39
    Thanked 92 Times in 48 Posts
    Quote Originally Posted by encode View Post
    My current image codec compresses such file to 4,508,344 bytes in 0.5 sec or less.
    This sounds very interesting! When will it be released?

    butterfly4.bmp was converted form camera RAW directly - i.e. no ex-JPEG.
    The ratio BMPsize/BMFsize is more often between 3:1 and 4:1 on such images, while on Butterfly4 it's almost 5:1.

    BTW, ZPAQ has a special configuration for BMP images. Did you choose not to test it because it's in an external file?

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

  9. #9
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts

    Talking

    Quote Originally Posted by Alexander Rhatushnyak View Post
    The ratio BMPsize/BMFsize is more often between 3:1 and 4:1 on such images, while on Butterfly4 it's almost 5:1.
    I guess it's due to a massive Bokeh around a butterfly.

    Quote Originally Posted by Alexander Rhatushnyak View Post
    BTW, ZPAQ has a special configuration for BMP images. Did you choose not to test it because it's in an external file?
    Just unable to correctly use those CFG files...

    Quote Originally Posted by Alexander Rhatushnyak View Post
    This sounds very interesting! When will it be released?
    Well, I've just started. I've never written specialized image coders. My GUI archivers PIMPLE and PIM have intended transformations on top of the main compression algorithms. PIM has the most advanced algorithm of all my programs - color space transformation + LPC + PPMd.
    Currently I'm just tuning the CM part on various images, and testing various transformations before CM part, including various LPC schemes. So it's hard to say the exact release date. The most complex part is the codec itself - it should be properly designed and tested - and I must achieve some performance - compression speed vs compression ratio - to be on par with current codecs. Time limit already defined - just because user must able to browse compressed images instantly, nearly with no delay. GUI part was designed, file structure - that will contain signature, version number and CRC - already designed too. The BMP file will be compressed as is - bit to bit identical to the original, including the header and extra information that image may contain. I'm thinking about the format/codec name. All in all I'm about to enter new world!

  10. #10
    Member kampaster's Avatar
    Join Date
    Apr 2010
    Location
    ->
    Posts
    55
    Thanks
    4
    Thanked 6 Times in 6 Posts
    Greetings encode!
    Concerning offers!
    Don't forget to let out the console version necessarily!
    Thanks!

  11. #11
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    If convert butterfly4.bmp to PNM, FLIC will compress it to 4,414,260 bytes. And FLIC is fast!
    SO, what is FLIC algorithm, in brief?

  12. #12
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    237
    Thanks
    39
    Thanked 92 Times in 48 Posts

    Arrow

    Quote Originally Posted by encode View Post
    Time limit already defined - just because user must able to browse compressed images instantly, nearly with no delay.
    Time limit may be defined if you specify
    (1) what's the acceptable delay: is it half second, one second, two seconds, or ?
    (2) what's the width*height*bpp of images that must be displayed with acceptable delays,
    (3) on what hardware: i.e. what are the CPU and memory speeds and L1/L2/L3 cache sizes.

    About FLIC:
    http://www.linkedin.com/groups/FLIC-...256.S.42748372
    Currently I may not describe the algorithm, even in brief, only in extremely brief.
    Last edited by Alexander Rhatushnyak; 22nd July 2011 at 23:01.

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

  13. #13
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Alexander Rhatushnyak View Post
    Time limit may be defined if you specify
    (1) what's the acceptable delay: is it half second, one second, two seconds, or ?
    (2) what's the width*height*bpp of images that must be displayed with acceptable delays,
    (3) on what hardware: i.e. what are the CPU and memory speeds and L1/L2/L3 cache sizes.
    (1) About one second
    (2) Full HD screen shots (1920x1080) and 12 to 16 megapixel images. 24 or 32 bits per pixel
    (3) Modern hardware - Intel Core i7 first gen, SandyBridge, and so on. On this hardware you must able to play latest 3D games

  14. #14
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Can I call this new format and program BIM? Is such image format exists?

  15. #15
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts

  16. #16
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    But no such image format, seems to be...

  17. #17
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    190
    Thanks
    8
    Thanked 62 Times in 33 Posts
    Quote Originally Posted by encode View Post
    BMP ->
    Color space transform (Optionally) -> Delta transform (Optionally) -> CM or BWT+CM
    -> Compressed BMP
    Is the color space transform a lossless RGB to YCbCr transformation?
    If I am not mistaken BMP uses the (A)RGB (Alpha Red Green Blue) order, have you considered to reorder the color components on each line?
    i.e. instead of having this in memory:
    ARGB ARGB ARGB ARGB... ARGB
    turn it into
    AAAA...A RRRR...R GGGG...G BBBB...B
    Context would be components of the same color (instead of whole pixels triplet), or you could try to use n-3, n-6... (or n-4, n-8...) as a context.

  18. #18
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by caveman View Post
    Is the color space transform a lossless RGB to YCbCr transformation?
    Yep, kind of.

    Quote Originally Posted by caveman View Post
    If I am not mistaken BMP uses the (A)RGB (Alpha Red Green Blue) order, have you considered to reorder the color components on each line?
    i.e. instead of having this in memory:
    ARGB ARGB ARGB ARGB... ARGB
    turn it into
    AAAA...A RRRR...R GGGG...G BBBB...B
    Context would be components of the same color (instead of whole pixels triplet), or you could try to use n-3, n-6... (or n-4, n-8...) as a context.
    32-bit BMPs have RGB + alpha channel. The codec will be mainly intended for 24-bit BMPs (32-bit BMPs will be supported too). I tried all your examples and much much more! Many things that should work in a head, did not worked completely in real life.

  19. #19
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,474
    Thanks
    26
    Thanked 121 Times in 95 Posts
    caveman:
    Or even better - use a lossless transformation from MS HD Photo: http://en.wikipedia.org/wiki/JPEG_XR...sion_algorithm to decompose RGB into three separate planes and also compress them separately.

  20. #20
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    What about zpaq -mbmp_j4 -b0 ?

  21. #21
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    237
    Thanks
    39
    Thanked 92 Times in 48 Posts
    Quote Originally Posted by encode View Post
    PIM has the most advanced algorithm of all my programs - color space transformation + LPC + PPMd.
    On LPCB files, PIM 2.90 stops compressing at 141% (if BMPs, ~173% if PPMs). Better than RAR and FreeArc, but not as good as UHARC. Compression took more than an hour.
    Last edited by Alexander Rhatushnyak; 23rd July 2011 at 08:02.

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

  22. #22
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Matt Mahoney View Post
    What about zpaq -mbmp_j4 -b0 ?
    C:\Test>zpaq -mbmp_j4 -b0 c butterly4.zpaq-bmp butterfly4.bmp
    Using model bmp_j4.cfg
    zpaqopt C:\Users\Ilia\AppData\Local\Temp\zpaqtmp1476_0

    C:\Test>g++ -O3 -march=native -s -DNDEBUG C:\Users\Ilia\AppData\Local\Temp\zpaqt
    mp1476_0.cpp c:\zpaq\zpaq.o c:\zpaq\libzpaq.o -Ic:\zpaq -o C:\Users\Ilia\AppData
    \Local\Temp\zpaqtmp1476_0.exe
    'g++' is not recognized as an internal or external command,
    operable program or batch file.

    Creating archive butterly4.zpaq-bmp.zpaq
    zpaq -h -mcolorpre r butterfly4.bmp C:\Users\Ilia\AppData\Local\Temp\zpaqtmp1476
    _1.out
    Using model colorpre.cfg
    zpaqopt C:\Users\Ilia\AppData\Local\Temp\zpaqtmp3684_0

    C:\Test>g++ -O3 -march=native -s -DNDEBUG C:\Users\Ilia\AppData\Local\Temp\zpaqt
    mp3684_0.cpp c:\zpaq\zpaq.o c:\zpaq\libzpaq.o -Ic:\zpaq -o C:\Users\Ilia\AppData
    \Local\Temp\zpaqtmp3684_0.exe
    'g++' is not recognized as an internal or external command,
    operable program or batch file.


    I need g++?!

  23. #23
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Alexander Rhatushnyak View Post
    On LPCB files, PIM 2.90 stops compressing at 141% (if BMPs, ~173% if PPMs). Better than RAR and FreeArc, but not as good as UHARC. Compression took more than an hour.
    PIM recognizes TIF, TGA and BMP only.
    It uses filters on top of the main compression algorithm - PPMd - i.e. with the same high-order PPM as for all other files - far from perfect for transformed data. Just writing a fully featured image codec was not the goal at that time.

    BWT, can you suggest a file from your test set, that will be the most suitable for testing and tuning?

  24. #24
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	bim.png 
Views:	426 
Size:	631.1 KB 
ID:	1620  

  25. #25
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    Quote Originally Posted by encode View Post
    I need g++?!
    For JIT optimization. If it fails then ZPAQ will still work but not as fast. You can get rid of the error message with -j0, or if you want JIT then configure zpaqopt.bat to use a different compiler according to the readme.txt in the zpaq301.zip source distribution.

  26. #26
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Well, tested new codec - and I'm into something. Mainly I'm comparing it to BMF2 in fast mode and FLIC, since my codec is fast!

    A small comparison on old Waterloo test, all files are converted to PNM:

    No code has to be inserted here.



    Alex, can you provide me a few reference 24-bit image files?

  27. #27
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Interesting to see so big differences going both ways.

  28. #28
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,474
    Thanks
    26
    Thanked 121 Times in 95 Posts
    encode:
    Could you provide timings and memory consumption?

  29. #29
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    237
    Thanks
    39
    Thanked 92 Times in 48 Posts
    Quote Originally Posted by encode View Post
    A small comparison on old Waterloo test, all files are converted to PNM:
    According to A.C.T. ( http://compression.ca/act/act-graphics.html ) ERI32 5.1fre packs this set better, and it was released in 2002.
    Quote Originally Posted by m^2 View Post
    Interesting to see so big differences going both ways.
    There are no algorithms for 8-bit images and for artificial images in FLIC 1.5.demo.
    Quote Originally Posted by encode
    Alex, can you provide me a few reference 24-bit image files?
    There are lots of images on http://www.photographyblog.com, I don't know which of them are more suitable for your needs.
    Last edited by Alexander Rhatushnyak; 25th July 2011 at 08:21.

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

  30. #30
    Member toi007's Avatar
    Join Date
    Jun 2011
    Location
    Lisbon
    Posts
    35
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Are you going to use color reduction? in some aprox of lossy format?
    well if its lossless dont check my off topic bmp lossy format taht failed by a lot
    Last edited by toi007; 28th July 2011 at 20:00. Reason: dont know if im a nag

Page 1 of 4 123 ... LastLast

Similar Threads

  1. FLIC - a new fast lossless image compressor
    By Alexander Rhatushnyak in forum Data Compression
    Replies: 25
    Last Post: 10th January 2013, 19:46
  2. Lossless image coders
    By Madgeniy in forum Data Compression
    Replies: 26
    Last Post: 11th July 2011, 10:06
  3. GraLIC - new lossless image compressor
    By Alexander Rhatushnyak in forum Data Compression
    Replies: 17
    Last Post: 29th November 2010, 21:27
  4. New lossless compressor for 24-bit images (3 channels, 8 bits per channel)
    By Alexander Rhatushnyak in forum Data Compression
    Replies: 28
    Last Post: 23rd September 2010, 02:43
  5. image compressors
    By maadjordan in forum Forum Archive
    Replies: 5
    Last Post: 13th August 2007, 10:28

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •