Precompression of Tiff Images

[Simon Berger]

Here is my real first contribution to this community.

Version 0.1 Alpha - Simon Berger - 14-04-09
Precompression of Tiff files encoded with packbits.
It precompresses only the first page of multi-page tiff files yet.

Usage: c|d <infile> <outfile>

I created lossless precompression of tiff files encoded with packbits (only). More maybe coming later. It has the following limitations.
To understand most of them look at the specification of Tiff.

1) Only the first of multi-page tiff files will be precompressed. Is it necessary to do more? Most multi-page files are faxes or other documents which should be compressed with CCITT 4.

2) Images separated by tiles instead of strips can't be precompressed but they seem to be used rarely. I found one very big example here. There are more very nice examples in other directories which are working.

3) The whole file will be loaded into memory and additionally the image data will be extracted in one block into memory.

If someone tries this program I am interested in all commentaries. Exceptionally if you get one of the following outputs (if precompression works or not).

Debug: Offsets type is SHORT.
Debug: Counts type is SHORT.

The program converts the tiff file into an almost valid tiff file without compression. But can't be viewed because some offsets and values that aren't necessary for precompression won't be changed.
It has no validation if the process worked or is byte-identical but it did with all files I tested.

I don't know if I will continue improving it or will fix any of these limitations but I will release the source code and/or send it to schnaader for precomp later.

At the end a warning that this is a alpha program and be careful of any data damage. There is no check if output file exists but shouldn't generate any other problem.

EDIT:
Here you get the needed Visual C++ 2008 Redistributable Package if you don't already have it.

EDIT2:
Updated to Version 0.2.1. See posting 25

EDIT3:
Updated to Version 0.3.0. See posting 28

EDIT4:
Updated to Version 0.5.0 and released the source code.
See posting 45

[nanoflooder]

The program converts the tiff file into an almost valid tiff file without compression. But can't be viewed because some offsets and values that aren't necessary for precompression won't be changed.

Great! Keep up the good work Precompression can be very useful in everyday compression practice.

[toffer]

So you actually decompress that PackBits format?

[Simon Berger]

Yes thats the only thing I do. Additional changing some offsets to get everything byte identical. That was the really "hard" part because the image data isn't in one block. Instead it is separated in many blocks of some rows spread almost all over the file (worst case).

I have the idea to store the image data in one block (did that in the first try and create a bmp header. Tiff also has a color palette for 1,4 and 8 bit images that could easily converted to bmp palette (stores rgb rrrr-gggg-bbbb instead of rgbrgbrgbrgb ).
This should help many compressors with a special bmp model.

[mhajicek]

Nice to hear that precompression of another format is beeing tried to develop.

On my system I'm not able to even execute this tiny program. Does it need some dll?

[Simon Berger]

Ahh, good that you mention it I forgot. It needs the VC++ 2008 runtime.

http://www.microsoft.com/downloads/d...displaylang=en

[lunaris]

Ahh, good that you mention it I forgot. It needs the VC++ 2008 runtime.

http://www.microsoft.com/downloads/d...displaylang=en

If possible, create a plugin for precomp based on your program

[Simon Berger]

That's not possible but as mentioned above I will sent the source code to schnaader and I am sure he will include it if he has time.

This has also nothing to do with what you quoted. That is a normal set of dll files every compiler needs. I should have compiled this application with VC++ 2005 because almost everyone has these files installed.
You can safely install it you could need it later anyway.

[pat357]

Thanks for the nice tool, Simon !
I tested it on a couple of TIF's but the program was not able to restore the images.
Maybe I did something wrong..
As I didn't have any packbits Tiff's on my system, I created a few starting with JPG.
I used the freeware Farstone Image Viewer (http://www.faststone.org/FSViewerDownload.htm) to convert the JPG's to Tiff with packbits compression.
One 2 files, the prog. faled to restore the TIFF (no errors) and crashed on other one. Maybe this viewer doesn't create images conform the Tif spec ?

Could you test it yourself with tiff's created by this this otherwise great Farstone Viewer ?

-edit-
I also tried JPG->TIF conversion using Photoshop CS4, but it doesn't allow me to specify PackBits compression for Tif's.

[Simon Berger]

Thanks for the information. I will try the tool you mentioned. Try IrfanView to convert them instead this should work.

EDIT:
Interesting. Recompression is "fine" but the file is heavily different. It seems like you can compress packbits in really different ways. I have no other explanation yet.

[pat357]

Thanks for the information. I will try the tool you mentioned. Try IrfanView to convert them instead this should work.

EDIT:
Interesting. Recompression is "fine" but the file is heavily different. It seems like you can compress packbits in really different ways. I have no other explanation yet.

The "restored" image is also heavily distorted though..

[Simon Berger]

Really? In my example it was exactly the same (visually). Is it a picture you could send me?
To FastStone tiff implementation. It is a very bad one. There are value settings in the header which doesn't need to because they are only default and also it puts the whole image in one block.
But I will try to fix it.

[mhajicek]

Tried on irfanview generated images - 24bpp and 8bpp images OK, 1bpp images (b/w) lead to crash.

[Simon Berger]

Indeed - Thank you. I looked at b/w pictures at the beginning but didn't fully understand how they are stored and forgot them at the end.
Is fixed for me now.
It's also the only change.

@pat357
I experimented a little bit with the compression of packbits. It's simple but nevertheless it can be very different. I use the encoding from libtiff which seems to be used most times.
I am still interested in fixing this or getting it to work. It would be important to get a picture that leads to a different/broken visual result.

[pat357]

@pat357
I am still interested in fixing this or getting it to work. It would be important to get a picture that leads to a different/broken visual result.

I uploaded it here : http://rapidshare.com/files/221633067/mytif.7z.html

[toffer]

I just wanted to download some TIFs randomly from the net but was unable to find some -.- ?!? Are there no sources?

Would some tests with IrfanView (conversion) be ok?

[Shelwien]

http://images.google.com/images?as_s...mages&aq=f&oq=

[pat357]

http://images.google.com/images?as_s...mages&aq=f&oq=

Thanks for the link !
It seems that most images are infact .JPG renamed to Tiff...

[toffer]

How did you get that (filetype? Guess it's through some weird navigation

[Simon Berger]

http://download.osgeo.org/geotiff/samples/

Very nice source. Yeah Tiff files are rare and thats the reason I won't invest too much time. On the other hand it's an interesting target.
I today begun to support large files and reduce memory usage and secondly did the first step for an eventual bmp stream.

Conversion with IrfanView should work every time.

[pat357]

How did you get that (filetype? Guess it's through some weird navigation

http://images.google.com/advanced_image_search?hl=en

[toffer]

I randomly picked a couple of tiffs, which turned out to be jpegs, so i used IrfanView. Guess some people think renaming is conversion...

The recompressor crashed on some fake-tiffs w/o any output (not complaining about the wrong signature) but worked for the tested tiffs. Guess it's cause you used IView, too?

[m^2]

I created some tiffs with XnView. Seems to work.

[Simon Berger]

No, at the beginning I only used tiff images I found on my hard drive, from friends or the Internet and they worked all.
Yeah I guess it has to be more robust against bad data but I at the moment don't know know to do it. I have to validate if decompression goes wrong or non typical header information found.

[Simon Berger]

Here is the next version that changes the whole data handling. The maximum memory usage for compression and decompression are now:

compression: Strip_Size (on normal files 1-20kb)
decompression: Uncompressed_Image_size

The input file isn't be buffered anymore. The decompression needs more memory because I save the image in one block at offset 8 of the file to try to add a bitmap header next. Maybe I separate image data and other information then decompression would have the same usage.

Second change is the added validation of a right filestructure. The header information have to be in the right order and all necessary fields have to be filled. This wasn't there in version 0.15 that leads to crashes.
Would be nice if you could retest @toffer or someone else.

It shouldn't lead to any crash. If it does please post detailed information


Something I forgot to mention. There are two magic-codes for Tiff files. II and MM. MM stores everything in the motorola byte order. This is also supported from the first version. There are some geotiff files on the URL I posted with this order.
The source could also be compiled for Big Endian systems.

EDIT:
Fixed a wrong allocation size which could produce a crash.

[pat357]

Here is the next version that changes the whole data handling. The maximum memory usage for compression and decompression are now:

Would it work for TIFF's created by FastStone Image Viewer ?

[Simon Berger]

Sorry forgot to write. No. I didn't change anything there. I will give a try after bitmap playing and tile- limitation elimination.
At the moment all compressors like 7-zip and also ccmx don't give a better compression the packbits on normal pictures. As simpler the pictures are as better will be the compression of 7z and others against packbits because it can't handle redundancy as good as they can.
Because of this I hope bmp header will boost it on compressors like ccmx.

[Simon Berger]

Version 0.3.0 out.

Changelog in short first.

+ Multi-page precompression
+ Validation of Recompression
+ Fixed more potential crashes
+ Reduced memory usage again

Multi-page precompression
In tiff files it is possible to create multi page documents. This way it could be use like pdf-files.
That's an important step to an eventual later CCITT 3 and 4 precompression which is used for textpictures.

Validation of Re-compression
Because of a complete rewrite it was easily possible to have some checks if re-compression success. There is no real comparing so changes are still possible. I won't do it at the moment because I hope this will end in precomp that will do the job.
The tiff-files produced from FastStone Image Viewer are now identified as not re-compressible for example.

Reduced memory usage again
compression is the same
decompression almost nothing now. Only one byte row width.
But additionally there are some bytes stored for every page.

The rewrite was needed which I realized before I implemented the adding of a bmp header. It was absolutely bad for changes.
The bmp test by hand was a success. paq produced much better results. But beginning with this release also 7z creates better results then the packed original file. .

If anyone is interested to implement any additional compression algorithm or something else let me now and I will send the source code to you. It will be out with the next version (bmp adding/converting) I hope.

Attached is the executable and one multi-page Tiff-file precompressed and packed with 7z. It's converted from CCITT 4 compression and found on the internet.

[Simon Berger]

"Bmp Conversion" is done for 16- and 24 bit images. 1, 2, 4 and 8 are on queue due to the color palette but that's not really a problem. Results are much better then I thought.

All files of the first test compressed with paq8p3 use switch 7. Second uses switch 5.

#First Test-file 1024x768. Packbits doesn't work well due to much details

original: 2,376,340
precompressed-bmp: 2,362,651 bytes
precompressed+bmp: 2,362,649 bytes
paq8p3+original: 1,008,408 bytes
paq8p3+precompressed-bmp: 975,990 bytes
paq8p3+precompressed+bmp: 659,243 bytes

#Second Test-file is the well known FlashMX.pdf converted to a multi-page TIFF and surely compressed with packjpg. It's not much better because the bmp model compresses only one bmp and the other one could use more data/redundancy I guess ?!?

precompressed+bmp: 587,895,156 bytes
original: 40,992,708 bytes
paq8p3+original: 2,840,049 bytes
paq8p3+precompressed+bmp: 2,578,709 bytes (90 bmp streams)

#Third Test file another one-page image. It's a psd high resolution resized image. I know those sort of pictures never will be saved as TIFF-files but still TIFF files could get much details and a high resolution. 4092x3069

precompressed: 37,699,877 bytes
original: 27,535,810 bytes
paq8p3+original: 5,342,787 bytes (1637.22 sec)
paq8p3+precompressed: 2,415,141 bytes (494.58 sec)

(Timings aren't absolutly right because I used the computer (no intensive use) but I started both the same time and the original had a lot of time alone at the end.

Does the paq model need DWORD padding? Then it's also limited to DWORD aligned pictures.
With this for the next release uncompressed images will get precompressed too.

Another idea is to convert images inside a multi-page document like FlashMX that are black-white only to 1bit. Image programs don't do this on there own.

EDIT:
Added test 3

[osmanturan]

Does the paq model need DWORD padding? Then it's also limited to DWORD aligned pictures.

This is not related to PAQ's model. It's an essential feature of an ordinal BMP file. All rows are DWORD aligned. And worst case is that "padding bytes" could be anything. But, usually they are all zeros. You can have look on wikipedia. There is a good documentation about BMP format.