PAQ8O released!

[LovePimple]

Andreas Morphis has made some improvements to the .bmp model in paq8n. The new program is paq8o. There is also a paq8osse.exe which requires SSE2 support (I think Pentium 4 or higher), which is a little faster, but archive compatible with paq8o.exe. Some tests with rafale.bmp:

634,547 66 sec with paq8n -5
551,665 78 sec with paq8o -5, 72 sec with paq8osse -5
(beats WinRK)

Non .bmp data should compress about the same as paq8n. I will run more tests but I expect paq8osse to be about 10% faster than paq8n on most data other than .bmp. The speedup is due to compiling with Intel C++.

http://cs.fit.edu/~mmahoney/compression/

Mirror: Download PAQ8O + bug fix

[Bulat Ziganshin]

erll, after Z they can use russian dictionary

and rafale isn't typical bmp. at least with my program it doesn't use MM compression

[LovePimple]

erll, after Z they can use russian dictionary

Good idea!

[LovePimple]

Quick (-7) test...

Test Machine: AMD Sempron 2400+

PAQ8O compressed bliss.bmp to 288,994 bytes in 61.25 seconds.

[Matt Mahoney]

and rafale isnt typical bmp. at least with my program it doesnt use MM compression

True, it is actually a 16 bit image that is "sign extended" at the wrong end. All the green values end with bits 000 or 111. All the red and blue values end with 0000 or 1111. If you set all the extra low bits to 0 (last 2 of green, last 3 of red and blue), then it improves the image quality and makes it more compressible like this:

635,470 rafale.bmp.paq8l
623,957 out.bmp.paq8l

I havent tried it with paq8o yet.

[Matt Mahoney]

It also works for paq8osse -5

551,665 rafale.bmp.paq8n
542,222 out.bmp.paq8n

Here is the program to drop the low bits of rafale.bmp to out.bmp

// drop low bits of rafale.bmp -> out.bmp
#include <stdio.h>
int main() {
FILE* in=fopen("rafale.bmp", "rb");
if (!in) {perror("rafale.bmp"); return 0;}
FILE* out=fopen("out.bmp", "wb");
for (int i=0; i<54; ++i) putc(getc(in), out); // copy header
int c;
while ((c=getc(in))!=EOF) {
putc(c&0xf8, out); // blue
putc(getc(in)&0xfc, out); // green
putc(getc(in)&0xf8, out); // red
}
return 0;
}

Here is the program to put them back. The output is identical to rafale.bmp.

// put back low bits of rafale.bmp (out.bmp -> rafale2.bmp)
#include <stdio.h>
int main() {
FILE* in=fopen("out.bmp", "rb");
if (!in) {perror("out.bmp"); return 0;}
FILE* out=fopen("rafale2.bmp", "wb");
for (int i=0; i<54; ++i) putc(getc(in), out); // copy header
int c;
while ((c=getc(in))!=EOF) {
putc(c+((c>>3)&1)*7, out); // blue
c=getc(in);
putc(c+((c>>2)&1)*3, out); // green
c=getc(in);
putc(c+((c>>3)&1)*7, out); // red
}
return 0;
}

[nimdamsk]

Matt, you are the true hacker Thanks!

P.S. If you'll change both "fc" and "f8" masks to "70" you'll get funny surrealistic picture, something like heat scanning And this works for any input picture

[LovePimple]

Thanks for the info Matt!

[roytam1]

it is actually a 16 bit image that is "sign extended" at the wrong end. All the green values end with bits 000 or 111. All the red and blue values end with 0000 or 1111.

for 16bit bmp bitmaps files, compression=3 (BI_BITFIELDS) and 3 bitmasks are placed in palette area.
so you should check the bit masks instead.

[Matt Mahoney]

rafale.bmp is stored as 24 bit image. There is no compression (value is 0) and no palette. I know it was originally a jpeg image, then a wallpaper on Werner's computer. When you make a jpeg a wallpaper, Windows converts it to .bmp. I think at some point it was converted to a 16 bit image, and then to 24 bit, but I don't know how. I have not seen this kind of artifact in any other images.

There is a smaller version of the original at http://www.defenseindustrydaily.com/cat/lockheed-m artin/page/33/
but I have not found the original.

[Piotr Tarsa]

PAQ8O compressed bliss.bmp to 288,994 bytes

btw:
winrar compresses bliss.bmp to 382 895 bytes (382 822 internally, without headers). you must select force text compression (ppm), set model order to three and select force truecolor bitmap compression in advanced options.

[LovePimple]

PAQ8O v2 has been released. PAQ8O v2 is basically the same as the unofficial bug fix version which was available from my home page a short time after PAQ8O v1 was first released.

http://cs.fit.edu/~mmahoney/compression/

Mirror: Download PAQ8O v2


The bug fix version is still available from my home page.

[Matt Mahoney]

I checked that LovePimple's paq8o bug fix is compatible with paq8o v2. Both versions of paq8osse.exe also work on my Athlon-64 3500+ and both are also compatible with paq8o.

[LovePimple]

Thanks for testing Matt!

[LovePimple]

We now wait for PAQ8P!

[Stephan Busch]

I am currently running PAQ8oSSE2 v2 on my reference system.
This is the 6th try to compress the bmp testset, but unfortunately my vista64 shuts down with bluescreen after 10 hours. the resulting archive size is 277 MB. Is there a way to check if PAQ8 did finish compressing before bluescreen incident? I mean, how can I verify that all bits of my bmp.tar are inside PAQ8o archive (apart from starting decompression)? Does PAQ8o give a warning or calculates CRC if archive is broken (because of my PC crash)?

I don't know if the crash was caused by PAQ8o, so I don't want to
produce panic.

[Piotr Tarsa]

We now wait for PAQ8P!

paq with lpc model for wave files would be good

theres over 10 mb of wav data on mfc testset. with lpc model paq should be able to shave one or two megabytes out of compressed file.

[LovePimple]

Good idea!

[Vacon]

Hello everyone,

for the interested audience:
http://sourceforge.net/projects/powerpaq/

Best regards!

[Matt Mahoney]

Is there a way to check if PAQ8 did finish compressing before bluescreen incident?

Not really without decompressing, but if the output file size is a multiple of 4096 then the file was probably not closed. Is the output the same size each time the computer crashes? Also, does it crash on paq8o (not sse)? It was compiled with g++ instead of Intel.

[Stephan Busch]

The crashes were produced by weak web connection and not by PAQ8oSSE. The tests are finished now and I will put the results online this night. I decompressed the file to verify and there were no errors. As expected, PAQ8o now leads bitmap compression with a winning entry sending WinRK to 2nd place.
If only a WAV filter could be included, PAQ8o would become this years number one. Is there also a chance to add ECM filter for disc image files (.IMG, .BIN)?

[maadjordan]

i second the ECM filter..and the wav filter ..

[Stephan Busch]

what means 'I second' ?, Maadjordan

[maadjordan]

i vote for your suggestion (ecm,wav) filters in paq8

[Stephan Busch]

cool

Let's see if our voice is heard by master Matt

[amorphis]

Non .bmp data should compress about the same as paq8n. I will run more tests but I expect paq8osse to be about 10% faster than paq8n on most data other than .bmp. The speedup is due to compiling with Intel C++.

There is a minor speedup other than compiling with Intel C++. Training is skipped if there is no error (err=0). The speedup depends on the randomness of the data being compressed. For text/log files paq8o is up to 8% faster than paq8n (both compiled with Mingw with the same compiler options).