FP8 (= Fast PAQ8)

[Jan Ondrus]

FP8 (= Fast PAQ8) is based on paq8px_v69 compressor. It has removed some models so it is faster on default data. It is only little bit slower than paq8pf (at least on my machine) while providing better compression ratio.
Special data (wav, bmp, jpeg) are compressed with original model (with same speed and ratio as paq8px)

speed:
- paq8px_v69 = 50.8s
- paq8pf = 6.0s
- fp8 = 6.6s

compression ratio (SFC -7):
- paq8px_v68 = 8764929 bytes
- paq8pf = 9402940 bytes
- fp8 = 9167064 bytes

[Skymmer]

Thanks Jan! Going to test it soon.

[Jan Ondrus]

Thanks Jan! Going to test it soon.

Can you please compare it with paq8pf and nanozip best mode?

[Wladmir]

Jan Ondrus, you could develop a new version of FP8 without using the compression model special data (wav, bmp, jpeg) of paq8px. That would make the FP8 even faster.

[schnaader]

Jan Ondrus, you could develop a new version of FP8 without using the compression model special data (wav, bmp, jpeg) of paq8px. That would make the FP8 even faster.

I think this is what paq8pf is doing and that this actually is the reason why FP8 was created: To give better compression ratio than paq8pf on special data, but still offering a speedup on default data.

So, have a look at paq8pf instead if you're looking for best speed on all kind of data.

EDIT: It seems that paq8pf has been stopped because of FP8 and the executable attachment has been removed, but I'm sure you can get a copy from someone (for example Jan).

[Jan Ondrus]

Jan Ondrus, you could develop a new version of FP8 without using the compression model special data (wav, bmp, jpeg) of paq8px. That would make the FP8 even faster.

Yes, but using default model for special data is not good idea (it would mean bad compression ratio for such data). I think special models should/can be made faster instead (at least as fast as default model in current version). This can be done by reducing number of contexts/predictions used by models.

[Jan Ondrus]

I think this is what paq8pf is doing

I don't think it is.

and that this actually is the reason why FP8 was created: To give better compression ratio than paq8pf on special data, but still offering a speedup on default data.

The reason was: Make paq version with same or simalar speed as paq8pf but with better compression ratio and with source code available.

EDIT:
paq8pf.exe

[Wladmir]

I made comparisons at the highest supported by my computer. (1 GByte DDR2).

method | level | size
none | 0 | 163.340 bytes
paq8px | -7 | 80.64s | 33.086bytes
fp8 | -7 | 9.77s | 35.840bytes
NanoZip | nz_cm | 1.96s | 37.190bytes
paq8pf | -7 | 8.92s | 37.370 bytes
paq9a | -8 | 3.19s | 38.730bytes

[Bulat Ziganshin]

I made comparisons at the highest supported by my computer.

what about lpaq? it may be faster and compress better than paq9

[Wladmir]

Jan Ondrus, could you create the version "fp8_v1_sse2.exe"? The version "paq8px_v69_sse2.exe" was quick on my computer.

[mhajicek]

Nice speedup without loosing too much.

i tried compressing ~3.5 GB (2 files), and it was ok, but the info displayed at the end was not:

....
1/2 Filename: cc.tar.001 (1887436800 bytes)
....
Compressed from 1887436800 to 1267174912 bytes.
2/2 Filename: cc.tar.002 (1559366144 bytes)
....
Compressed from 1559366144 to -1267174961 bytes.
....
Total -848164352 bytes compressed to -1 bytes.
Time 50797.76 sec, used 1515851009 bytes of memory

[moisesmcardona]

Nice speedup without loosing too much.

i tried compressing ~3.5 GB (2 files), and it was ok, but the info displayed at the end was not:

....
1/2 Filename: cc.tar.001 (1887436800 bytes)
....
Compressed from 1887436800 to 1267174912 bytes.
2/2 Filename: cc.tar.002 (1559366144 bytes)
....
Compressed from 1559366144 to -1267174961 bytes.
....
Total -848164352 bytes compressed to -1 bytes.
Time 50797.76 sec, used 1515851009 bytes of memory

The problem was that you tried compressing more than 2GB. PAQ Compressors can handle files and folders with a total of 2GB or less space

[mhajicek]

moisesmcardona:
I know about this limit, but it is imho a single file limit (these files are immediately skipped), not limit of total amount of data to be compressed, at least my 2,5GB archive (3,5GB dec.) works perfectly. I just reported the incorrect numbers diplayed in console window, probably some counters are not designed for so big archives.

[Matt Mahoney]

That problem is fixed in zpaq and zp. There is no file size limit.

[Jan Ondrus]

fp8_v2 version with minor changes in model compared to fp8_v1

[Matt Mahoney]

Nice improvement on the Silesia corpus. BTW I had to compress ooffice on the D: drive because of the g++ tmpfile() bug. It only happens when .exe is detected.

http://mattmahoney.net/dc/silesia.html

[Matt Mahoney]

#8 on LTCB. I decided to give it its own entry since it is a fork of paq8. http://mattmahoney.net/dc/text.html#1544

[qqwertyy]

I found bug in fp8* code or in g++/glibc or in my g++ options knowledge :

0)my Linux box

0a)x86_64:
Linux mobilestar 2.6.32.59 #1 SMP Mon Mar 19 20:27:43 GMT 2012 x86_64 Intel(R) Pentium(R) Dual CPU T2370 @ 1.73GHz GenuineIntel GNU/Linux
gcc-g++-4.2.3-x86_64-1
glibc-2.7-x86_64-12ttt

0b)i686:
Linux mobilestar 2.6.32.59 #1 SMP PREEMPT Mon Mar 19 16:48:50 UTC 2012 i686 Intel(R) Pentium(R) Dual CPU T2370 @ 1.73GHz GenuineIntel GNU/Linux
gcc-g++-4.2.3-i486-1
glibc-2.7-i486-12_slack12.1


1)

1a)x86_64(pure):
g++ -O2 -static -fPIC -DUNIX -DNOASM -fomit-frame-pointer fp8.cpp -o fp8.x86_64

1b)i686:
g++ -O2 -static -march=i686 -DUNIX -DNOASM -fomit-frame-pointer fp8.cpp -o fp8.i686


2)after successful compression on pure i686 or pure x86_64 via command:
fp8_v2 -6 Modes

i have to different files Modes.fp8 and some members of Modes.fp8 cannot be unpacked
on the system with architecture different architecture:

2a) unpacking x86_64 archive (packed on x86_64 via x86_64 executable ) on i686 via i686 executable :
http://pastebin.com/rY0vKnzH

2b) unpacking i686 archive (packed on i686 via i686 executable ) on x86_64 via x86_64 executable :
http://pastebin.com/6G54Rj9p

2c) unpacking i686 archive (packed on i686 via i686 executable ) on x86_64 via the same i686 executable :
http://pastebin.com/Khyp7MYT

The same history with fp8_v1 (but on the members "./Modes/PSKdesc.htm" and "./Modes/Throb1X.png")

P.S.
I`m sorry for my broken english.

[qqwertyy]

http://zalil.ru/33069253

c9a90eb6b7ff09b175340c8ce1ea87fb *fp8-bug.zip

Test files set:

http://zalil.ru/33069268

9c79ef0f6bf5e133aaabb8f81f2a6e66 *Modes.7z

[Jan Ondrus]

It is known that different compiles produce different archives when compressing WAV/audio files in paq8px and fp8 programs. It is because fp8 and paq8px contain wavmodel by Florin Ghido which uses Floating-point arithmetic for its matrix computations. I was unable to rewrite it with Integer Arithmetic while providing same compression ratio. Maybe someone more educated can do it.

[Shelwien]

I tried adding this - http://nishi.dreamhosters.com/u/double.inc
and compression really drops when __int64 is used for internal representation (and a little when float is used too),
while it stays the same with double.
But I think it should be possible to add manual handling of floating point there, and get a working integer port.

[Stephan Busch]

Hey Jan,

I have done decompression tests on my testsets. 10 of 11 sets could be decompressed properly (MD5 checksums matched) but the sources.tar.fp8 could not be decompressed error-free. Can you please have a look on it?

[Jan Ondrus]

bug fix in jpegmodel
should be a little slower with a little better compression ratio.

[Matt Mahoney]

Improved compression ratio on the Silesia corpus. http://mattmahoney.net/dc/silesia.html

[Skymmer]

should be a little slower with a little better compression ratio.

Jan, thanks for new version.
Quick and dirty test on 30 jpeg files with total size of 75 873 544 bytes. Performed on i5-2500K @ 3700 MHz.

original   -------     75 873 544
fp8_v2     657.281     57 015 208
fp8_v3     671.625     57 029 037

[Matt Mahoney]

Compression is also improved on LTCB from .1544 to .1532 using 8% more time. It is still in 8'th place.
http://mattmahoney.net/dc/text.html#1532

[questor]

a small note: I've tried to build a library out of fp8_v2 to be able to use in own programs, but without success. when I compress a file and in the same run decompress the same file it's not depacked correctly. I've changed many static variables to use dynamic allocated ones and reinitialised the global variables, but with this configuration I've got bufferover- or underruns. I've tried running only the decompressor with a valid compressed file and it generated the correct depacked file, but I've still got bufferoverruns.
I might have introduced them, but I think you doesn't get the debug-message because most of your buffers are static ones and they are not checked by the debug runtime.
and even after using dynamic alloced structures and filling the buffers with zeros the depacked content was only correct (only when compression the file before trying to decompress the file) for the first 20bytes and afterwards all was lost... but perhaps this is also the problem why different compiled programs will generate different compressed versions or your problems with the doubles. perhaps the compressed output is dependent on garbage in the memory. has anyone tried to compress a file on one os and decompress the file on a different os?

I might have introduced the errors with my changes, so be careful with my findings...

[Shelwien]

it may be related to the already mentioned floating-point problem.
If encoder and decoder are compiled separately and in different environments, its quite possible that they won't be compatible.
Try using strict fp options for compiler.

[Nania Francesco]

I want to congratulate the author of this fantastic compressor that you can still improve a lot. I just wanted to say that I should simplify the code to get better in less time and that the road taken in later versions is perhaps not the right one!
Good job !
Best Regards !

[Matt Mahoney]

The variables are static because they need to retain their values between function calls. If you are trying to make it thread safe, then you have a lot of work to do.