Page 1 of 2 12 LastLast
Results 1 to 30 of 53

Thread: Precompression of Tiff Images

  1. #1
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts

    Precompression of Tiff Images - Update: 0.5.0

    Here is my real first contribution to this community.

    Code:
    Version 0.1 Alpha - Simon Berger - 14-04-09
    Precompression of Tiff files encoded with packbits.
    It precompresses only the first page of multi-page tiff files yet.
    
    Usage: c|d <infile> <outfile>
    I created lossless precompression of tiff files encoded with packbits (only). More maybe coming later. It has the following limitations.
    To understand most of them look at the specification of Tiff.

    1) Only the first of multi-page tiff files will be precompressed. Is it necessary to do more? Most multi-page files are faxes or other documents which should be compressed with CCITT 4.

    2) Images separated by tiles instead of strips can't be precompressed but they seem to be used rarely. I found one very big example here. There are more very nice examples in other directories which are working.

    3) The whole file will be loaded into memory and additionally the image data will be extracted in one block into memory.

    If someone tries this program I am interested in all commentaries. Exceptionally if you get one of the following outputs (if precompression works or not).

    Code:
    Debug: Offsets type is SHORT.
    Debug: Counts type is SHORT.
    The program converts the tiff file into an almost valid tiff file without compression. But can't be viewed because some offsets and values that aren't necessary for precompression won't be changed.
    It has no validation if the process worked or is byte-identical but it did with all files I tested.

    I don't know if I will continue improving it or will fix any of these limitations but I will release the source code and/or send it to schnaader for precomp later.

    At the end a warning that this is a alpha program and be careful of any data damage. There is no check if output file exists but shouldn't generate any other problem.

    EDIT:
    Here you get the needed Visual C++ 2008 Redistributable Package if you don't already have it.

    EDIT2:
    Updated to Version 0.2.1. See posting 25

    EDIT3:
    Updated to Version 0.3.0. See posting 28

    EDIT4:
    Updated to Version 0.5.0 and released the source code.
    See posting 45
    Last edited by Simon Berger; 4th May 2009 at 02:41.

  2. #2
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    215
    Thanks
    0
    Thanked 0 Times in 0 Posts
    The program converts the tiff file into an almost valid tiff file without compression. But can't be viewed because some offsets and values that aren't necessary for precompression won't be changed.
    Great! Keep up the good work Precompression can be very useful in everyday compression practice.

  3. #3
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    So you actually decompress that PackBits format?

  4. #4
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Yes thats the only thing I do. Additional changing some offsets to get everything byte identical. That was the really "hard" part because the image data isn't in one block. Instead it is separated in many blocks of some rows spread almost all over the file (worst case).

    I have the idea to store the image data in one block (did that in the first try and create a bmp header. Tiff also has a color palette for 1,4 and 8 bit images that could easily converted to bmp palette (stores rgb rrrr-gggg-bbbb instead of rgbrgbrgbrgb ).
    This should help many compressors with a special bmp model.
    Last edited by Simon Berger; 15th April 2009 at 00:01.

  5. #5
    Member
    Join Date
    Mar 2009
    Location
    Prague, CZ
    Posts
    60
    Thanks
    27
    Thanked 6 Times in 6 Posts
    Nice to hear that precompression of another format is beeing tried to develop.

    On my system I'm not able to even execute this tiny program. Does it need some dll?

  6. #6
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Ahh, good that you mention it I forgot. It needs the VC++ 2008 runtime.

    http://www.microsoft.com/downloads/d...displaylang=en
    Last edited by Simon Berger; 15th April 2009 at 00:06.

  7. #7
    Member
    Join Date
    May 2008
    Location
    brazil
    Posts
    163
    Thanks
    0
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by Simon Berger View Post
    Ahh, good that you mention it I forgot. It needs the VC++ 2008 runtime.

    http://www.microsoft.com/downloads/d...displaylang=en
    If possible, create a plugin for precomp based on your program

  8. #8
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    That's not possible but as mentioned above I will sent the source code to schnaader and I am sure he will include it if he has time.

    This has also nothing to do with what you quoted. That is a normal set of dll files every compiler needs. I should have compiled this application with VC++ 2005 because almost everyone has these files installed.
    You can safely install it you could need it later anyway.
    Last edited by Simon Berger; 15th April 2009 at 00:33.

  9. #9
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Thanks for the nice tool, Simon !
    I tested it on a couple of TIF's but the program was not able to restore the images.
    Maybe I did something wrong..
    As I didn't have any packbits Tiff's on my system, I created a few starting with JPG.
    I used the freeware Farstone Image Viewer (http://www.faststone.org/FSViewerDownload.htm) to convert the JPG's to Tiff with packbits compression.
    One 2 files, the prog. faled to restore the TIFF (no errors) and crashed on other one. Maybe this viewer doesn't create images conform the Tif spec ?

    Could you test it yourself with tiff's created by this this otherwise great Farstone Viewer ?

    -edit-
    I also tried JPG->TIF conversion using Photoshop CS4, but it doesn't allow me to specify PackBits compression for Tif's.
    Last edited by pat357; 15th April 2009 at 01:00.

  10. #10
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Thanks for the information. I will try the tool you mentioned. Try IrfanView to convert them instead this should work.

    EDIT:
    Interesting. Recompression is "fine" but the file is heavily different. It seems like you can compress packbits in really different ways. I have no other explanation yet.
    Last edited by Simon Berger; 15th April 2009 at 01:41.

  11. #11
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by Simon Berger View Post
    Thanks for the information. I will try the tool you mentioned. Try IrfanView to convert them instead this should work.

    EDIT:
    Interesting. Recompression is "fine" but the file is heavily different. It seems like you can compress packbits in really different ways. I have no other explanation yet.
    The "restored" image is also heavily distorted though..

  12. #12
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Really? In my example it was exactly the same (visually). Is it a picture you could send me?
    To FastStone tiff implementation. It is a very bad one. There are value settings in the header which doesn't need to because they are only default and also it puts the whole image in one block.
    But I will try to fix it.

  13. #13
    Member
    Join Date
    Mar 2009
    Location
    Prague, CZ
    Posts
    60
    Thanks
    27
    Thanked 6 Times in 6 Posts
    Tried on irfanview generated images - 24bpp and 8bpp images OK, 1bpp images (b/w) lead to crash.

  14. #14
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Indeed - Thank you. I looked at b/w pictures at the beginning but didn't fully understand how they are stored and forgot them at the end.
    Is fixed for me now.
    It's also the only change.

    @pat357
    I experimented a little bit with the compression of packbits. It's simple but nevertheless it can be very different. I use the encoding from libtiff which seems to be used most times.
    I am still interested in fixing this or getting it to work. It would be important to get a picture that leads to a different/broken visual result.
    Attached Files Attached Files

  15. #15
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    @pat357
    I am still interested in fixing this or getting it to work. It would be important to get a picture that leads to a different/broken visual result.
    I uploaded it here : http://rapidshare.com/files/221633067/mytif.7z.html

  16. #16
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I just wanted to download some TIFs randomly from the net but was unable to find some -.- ?!? Are there no sources?

    Would some tests with IrfanView (conversion) be ok?

  17. #17
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,375
    Thanks
    214
    Thanked 1,023 Times in 544 Posts

  18. #18
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Thanks for the link !
    It seems that most images are infact .JPG renamed to Tiff...

  19. #19
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    How did you get that (filetype? Guess it's through some weird navigation

  20. #20
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    http://download.osgeo.org/geotiff/samples/

    Very nice source. Yeah Tiff files are rare and thats the reason I won't invest too much time. On the other hand it's an interesting target.
    I today begun to support large files and reduce memory usage and secondly did the first step for an eventual bmp stream.

    Conversion with IrfanView should work every time.

  21. #21
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by toffer View Post
    How did you get that (filetype? Guess it's through some weird navigation
    http://images.google.com/advanced_image_search?hl=en

  22. #22
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I randomly picked a couple of tiffs, which turned out to be jpegs, so i used IrfanView. Guess some people think renaming is conversion...

    The recompressor crashed on some fake-tiffs w/o any output (not complaining about the wrong signature) but worked for the tested tiffs. Guess it's cause you used IView, too?

  23. #23
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    I created some tiffs with XnView. Seems to work.

  24. #24
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    No, at the beginning I only used tiff images I found on my hard drive, from friends or the Internet and they worked all.
    Yeah I guess it has to be more robust against bad data but I at the moment don't know know to do it. I have to validate if decompression goes wrong or non typical header information found.

  25. #25
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Here is the next version that changes the whole data handling. The maximum memory usage for compression and decompression are now:

    compression: Strip_Size (on normal files 1-20kb)
    decompression: Uncompressed_Image_size

    The input file isn't be buffered anymore. The decompression needs more memory because I save the image in one block at offset 8 of the file to try to add a bitmap header next. Maybe I separate image data and other information then decompression would have the same usage.

    Second change is the added validation of a right filestructure. The header information have to be in the right order and all necessary fields have to be filled. This wasn't there in version 0.15 that leads to crashes.
    Would be nice if you could retest @toffer or someone else.

    It shouldn't lead to any crash. If it does please post detailed information


    Something I forgot to mention. There are two magic-codes for Tiff files. II and MM. MM stores everything in the motorola byte order. This is also supported from the first version. There are some geotiff files on the URL I posted with this order.
    The source could also be compiled for Big Endian systems.

    EDIT:
    Fixed a wrong allocation size which could produce a crash.
    Attached Files Attached Files
    Last edited by Simon Berger; 18th April 2009 at 22:42.

  26. #26
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by Simon Berger View Post
    Here is the next version that changes the whole data handling. The maximum memory usage for compression and decompression are now:
    Would it work for TIFF's created by FastStone Image Viewer ?

  27. #27
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Sorry forgot to write. No. I didn't change anything there. I will give a try after bitmap playing and tile- limitation elimination.
    At the moment all compressors like 7-zip and also ccmx don't give a better compression the packbits on normal pictures. As simpler the pictures are as better will be the compression of 7z and others against packbits because it can't handle redundancy as good as they can.
    Because of this I hope bmp header will boost it on compressors like ccmx.

  28. #28
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Version 0.3.0 out.

    Changelog in short first.

    Code:
    + Multi-page precompression
    + Validation of Recompression
    + Fixed more potential crashes
    + Reduced memory usage again
    Multi-page precompression
    In tiff files it is possible to create multi page documents. This way it could be use like pdf-files.
    That's an important step to an eventual later CCITT 3 and 4 precompression which is used for textpictures.

    Validation of Re-compression
    Because of a complete rewrite it was easily possible to have some checks if re-compression success. There is no real comparing so changes are still possible. I won't do it at the moment because I hope this will end in precomp that will do the job.
    The tiff-files produced from FastStone Image Viewer are now identified as not re-compressible for example.

    Reduced memory usage again
    compression is the same
    decompression almost nothing now. Only one byte row width.
    But additionally there are some bytes stored for every page.

    The rewrite was needed which I realized before I implemented the adding of a bmp header. It was absolutely bad for changes.
    The bmp test by hand was a success. paq produced much better results. But beginning with this release also 7z creates better results then the packed original file. .

    If anyone is interested to implement any additional compression algorithm or something else let me now and I will send the source code to you. It will be out with the next version (bmp adding/converting) I hope.

    Attached is the executable and one multi-page Tiff-file precompressed and packed with 7z. It's converted from CCITT 4 compression and found on the internet.
    Attached Files Attached Files
    Last edited by Simon Berger; 24th April 2009 at 05:20.

  29. #29
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    "Bmp Conversion" is done for 16- and 24 bit images. 1, 2, 4 and 8 are on queue due to the color palette but that's not really a problem. Results are much better then I thought.

    All files of the first test compressed with paq8p3 use switch 7. Second uses switch 5.

    #First Test-file 1024x768. Packbits doesn't work well due to much details
    Code:
    original: 2,376,340
    precompressed-bmp: 2,362,651 bytes
    precompressed+bmp: 2,362,649 bytes
    paq8p3+original: 1,008,408 bytes
    paq8p3+precompressed-bmp: 975,990 bytes
    paq8p3+precompressed+bmp: 659,243 bytes
    #Second Test-file is the well known FlashMX.pdf converted to a multi-page TIFF and surely compressed with packjpg. It's not much better because the bmp model compresses only one bmp and the other one could use more data/redundancy I guess ?!?
    Code:
    precompressed+bmp: 587,895,156 bytes
    original: 40,992,708 bytes
    paq8p3+original: 2,840,049 bytes
    paq8p3+precompressed+bmp: 2,578,709 bytes (90 bmp streams)
    #Third Test file another one-page image. It's a psd high resolution resized image. I know those sort of pictures never will be saved as TIFF-files but still TIFF files could get much details and a high resolution. 4092x3069
    Code:
    precompressed: 37,699,877 bytes
    original: 27,535,810 bytes
    paq8p3+original: 5,342,787 bytes (1637.22 sec)
    paq8p3+precompressed: 2,415,141 bytes (494.58 sec)
    
    (Timings aren't absolutly right because I used the computer (no intensive use) but I started both the same time and the original had a lot of time alone at the end.
    Does the paq model need DWORD padding? Then it's also limited to DWORD aligned pictures.
    With this for the next release uncompressed images will get precompressed too.

    Another idea is to convert images inside a multi-page document like FlashMX that are black-white only to 1bit. Image programs don't do this on there own.

    EDIT:
    Added test 3
    Last edited by Simon Berger; 25th April 2009 at 00:21.

  30. #30
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Simon Berger View Post
    Does the paq model need DWORD padding? Then it's also limited to DWORD aligned pictures.
    This is not related to PAQ's model. It's an essential feature of an ordinal BMP file. All rows are DWORD aligned. And worst case is that "padding bytes" could be anything. But, usually they are all zeros. You can have look on wikipedia. There is a good documentation about BMP format.
    BIT Archiver homepage: www.osmanturan.com

Page 1 of 2 12 LastLast

Similar Threads

  1. Images PreProcessor - PrePNG
    By PAQer in forum Data Compression
    Replies: 3
    Last Post: 21st May 2010, 13:21
  2. Idea for raising compression efficiency on disk images
    By Mexxi in forum Data Compression
    Replies: 10
    Last Post: 18th February 2010, 05:56
  3. Replies: 1
    Last Post: 13th May 2009, 11:46
  4. Compression used in .wim (Vista) images
    By jaclaz in forum Data Compression
    Replies: 9
    Last Post: 29th August 2008, 19:05

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •