paq8px

[Skymmer]

LovePimple, sorry for asking, but could you make PAQ8px_v5 compilation optimized for executable size, if it is not too difficult ?

[m^2]

Just a compilation, do further optimizations yourself.

[Skymmer]

Just a compilation, do further optimizations yourself.

Damn, you read my thoughts
Big thanx for compilation

[Stephan Busch]

PAQ8px_speed_optimised v4 runs flawlessly over my bitmap and cd image sets. thanks again for fixing, Jan and Kaido.

[Jan Ondrus]

paq8px_v6
-fixed: compressing using -8 switch was broken
-fixed: you could use (not working) -9 switch in previous versions

[LovePimple]

Thanks Jan!

Compiled...

EDIT: Attachment "paq8px_v6.zip" removed.

[Stephan Busch]

why has the paq8px_v5.zip attachment been removed?

This version runs over my bitmap and cd image sets like a charm
and seems to enter the top rank on Squeeze Chart.

So far, the cross-platform binary set compresses as follows:

paq8px_v4 -7 = 20.629.710 bytes
paq8px_v5 -7 = 20.571.295 bytes

very well done, guys.

[LovePimple]

why has the paq8px_v5.zip attachment been removed?

Because paq8px_v6 is exactly the same apart from a minor bug fix.

paq8px_v6
-fixed: compressing using -8 switch was broken
-fixed: you could use (not working) -9 switch in previous versions

[Simon Berger]

Same fix as done for paqp3.
Bmp detection for negative height was broken

See paq8p3 posting for details.

I miss the nice statistics from paq8p3 in paq8px.

Type    Size            Parts
unkn    480566          1
bmp8    512             1
Total parts: 2, fragmentation cost 10 bytes

[LovePimple]

Thanks Simon!

Compiled...

EDIT: Attachment "paq8px_v7.zip" removed.

[Jan Ondrus]

int height=i4(32);
if (height<0) {  // height negative -> image saved top-down 
  height*=(-1);
}

can be replaced with

int height=abs(i4(32));

paq8px_v8
- some code modified in 1-bit/8-bit/24-bit BMP detection (compression size is different only for 1-bit BMPs = was wrong starting offset in old version)

[Simon Berger]

Yeah you are right. Better solution.

I worked the first time with paq and was surprised to see a tiff model. On the other hand it is half implemented and only uses size and time.
I could finish it but this implementation has the following big limitations:

1) It needs to have all image parts concatenated (in the right order and without data between)
2) Only dword aligned images can be really well compressed with the bmpmodel.

Further there is a stream detection in "detection" which will be passed to bmpModel for example. There it will be detected again and called every x bytes with a possible re-detection of a bmp stream inside this stream.

Is this only to skip the headers? I don't have an overview of the whole project so I am going to discuss before doing anything but can't this be reformated to skip the header in "detect" and call bmpModel on the exact image start offset?

[kaitz]

I noticed that IrfanView creats tiff images where signature is at the beginning then offset to image description table witch is at the end of image data.
paq8 image model cant handle it. If transformig it after detection so that image data is at the end it can work.

When paq8 compresses tiff image it needs width of it but it is after image data so it is problem. Reorderering data in detected image is solution for me i think.

[LovePimple]

int height=i4(32);
if (height<0) {  // height negative -> image saved top-down 
  height*=(-1);
}

can be replaced with

int height=abs(i4(32));

paq8px_v8
- some code modified in 1-bit/8-bit/24-bit BMP detection (compression size is different only for 1-bit BMPs = was wrong starting offset in old version)

Thanks Jan!

Compiled...

EDIT: Attachment "paq8px_v8.zip" removed.

[Simon Berger]

Tiff is a too complex file format to create a small model. There indeed would be a step before required. My TiffPreprocession is exactly what could be implemented.
But I have no idea how to easily do this.

[Jan Ondrus]

My idea is to replace current block types (BMPFILE1, BMPFILE4, BMPFILE8, BMPFILE24, TIFFFILE, PGMFILE, PPMFILE, PBMFILE, RGBFILE) by IMAGEHDR, IMAGE24, IMAGE8 and IMAGE1.

Headers would be detected as IMAGEHDR block and raw image data as IMAGE1, IMAGE8 or IMAGE24 block. Width of image (needed for models) would be saved in compressed stream for every IMAGE1, IMAGE8 and IMAGE24 block.
This would avoid parsing headers of images twice (once for block segmenting and again for modeling)

What do you think?

[Jan Ondrus]

Further there is a stream detection in "detection" which will be passed to bmpModel for example. There it will be detected again and called every x bytes with a possible re-detection of a bmp stream inside this stream.

Is this only to skip the headers?

This is for
-finding raw data (skipping header)
-detecting width of image

some ideas how to avoid this are in previous post

[Simon Berger]

Yes thats something I thought about too but I there are still some needed knowing limitations of the source code. I yesterday began to implement tga24. That could be easily integrated in this step. The only big problem is a missing identification. It has no magic code and no version information. I used some fields that should be 0 to avoid many false detections.

Your idea should be done for all models (wav...). It would speed up compression by some percents I guess.

[Simon Berger]

This is for
-finding raw data (skipping header)
-detecting width of image
some ideas how to avoid this are in previous post

For the tga start I passed the fieldtype to bmpModel. In this case I would know how many bytes to skip. The could also be some extra params passed from detection.

The really bad situation is that in a block where bmp is progressed new compares with BM will be done (and more if BM found in image data).

[Jan Ondrus]

My idea is to replace current block types (BMPFILE1, BMPFILE4, BMPFILE8, BMPFILE24, TIFFFILE, PGMFILE, PPMFILE, PBMFILE, RGBFILE) by IMAGEHDR, IMAGE24, IMAGE8 and IMAGE1.

Headers would be detected as IMAGEHDR block and raw image data as IMAGE1, IMAGE8 or IMAGE24 block. Width of image (needed for models) would be saved in compressed stream for every IMAGE1, IMAGE8 and IMAGE24 block.
This would avoid parsing headers of images twice (once for block segmenting and again for modeling)

implemented
-now there are only im24bitModel, im8bitModel and im1bitModel instead of im24bitModel, pgmModel, bmpModel8, rgbModel8, bmpModel1 and pbmModel
-all image header detection is removed from models (only place for image detection is detect procedure)
-every image is divided into 2 blocks - IMAGEHDR (header) and IMAGEx (where x is 1, 8 or 24)
-there are additional 8 bytes in compressed stream (for size of IMAGEHDR block and for image width)
-experimental improved tiff detection (should detect uncompressed tiff created by IrfanView)
-RGB detection can detect images with more chanels (3 or 4) - chanels are compressed independently using im8bitModel (IMAGE8 block)
-during compression can't be printed such info BMP(width x height), instead of it program prints info about blocks: blocktype(size in bytes)

[schnaader]

Nice work! Tested on a small 320*240*24 bit test picture:

Original BMP        230,454 bytes
Original TIFF       230,866 bytes
BMP paq8px_v8 -4     63,077 bytes
BMP paq8px_v9 -4     63,083 bytes
TIFF paq8px_v8 -4    93,407 bytes
TIFF paq8px_v9 -4    62,782 bytes

TIFF compression is slightly better than BMP compression because the image data is stored upside down in one of them, so it shouldn't be better in general (except for padded BMPs perhaps).

[LovePimple]

Thanks Jan!

Compiled...

EDIT: Attachment "paq8px_v9.zip" removed.

[Simon Berger]

That was really fast! Thanks Jan. Will have a look at

[kaitz]

implemented
-now there are only im24bitModel, im8bitModel and im1bitModel instead of im24bitModel, pgmModel, bmpModel8, rgbModel8, bmpModel1 and pbmModel
-all image header detection is removed from models (only place for image detection is detect procedure)
-every image is divided into 2 blocks - IMAGEHDR (header) and IMAGEx (where x is 1, 8 or 24)
-there are additional 8 bytes in compressed stream (for size of IMAGEHDR block and for image width)
-experimental improved tiff detection (should detect uncompressed tiff created by IrfanView)
-RGB detection can detect images with more chanels (3 or 4) - chanels are compressed independently using im8bitModel (IMAGE8 block)
-during compression can't be printed such info BMP(width x height), instead of it program prints info about blocks: blocktype(size in bytes)

This reduces the problems in future management of images and removes the second processing from models.
I saw in file types list TEXT. Are you planing support and separate text like in paq8p3 version.

And what are future plans?

[Simon Berger]

There absolutely has to be some kind of merge between px and p3. PAQ stands for maximum compression but what kaitz did for many data types was very good and speed is too much better then px.

The tiff model seems to work really nice. I added a validation that the directory isn't on offset 8 then the probability for image block starting there is good.
Second I refined detection of 8+24-bit and added 1-bit.

[Jan Ondrus]

And what are future plans?

Don't like plans, but good ideas for future are:
-do same thing like for images also for audio (wav model)
-merging paq8p3 and paq8px

Are you planing support and separate text like in paq8p3 version.

no.

[kaitz]

kaitz wrote:
Are you planing support and separate text like in paq8p3 version.

no.

Maximum compression man.

[Darek]

Are you plan to detect TIFF's files with image header on the end of file?
Darek

[Simon Berger]

Here is paq8px_v10

Changes are:

Added 24-bit tga compression
Added More checks for Tiff
Added 1-bit Tiff compression
Changed many if[...]if[...] to if[...]else if[...] in detect
Fixed compiling error with MSVC++ because of unknown abs type

Two benchmarks for tga compression. I changed one byte at the beginning to disable tga detection.

original1: 5,242,924 bytes 1280x1024 (~1mb no picture data)
no tga1: 1,982,346 570.78 sec
tga detected1: 1,727,256 201.09 sec

original2: 9,437,228 bytes 2048x1536
no tga2: 566,218 769.51 sec
tga detected2: 468,732 99.83 sec

@Darek
The image header is at the end of the file for the files that are supported. It would be possible to add tiff files where the headers are in front of the image data but this would need more file seeking and header reading. Second problem are separated header and a higher possibility for non image data blocks in the assumed image only block .

[LovePimple]

Thanks Simon!

Compiled...

EDIT: Attachment "paq8px_v10.zip" removed.