PAQ8K

**Darek** · 3rd August 2009, 00:01

Hi! Thanks LovePimple for these speedup version!

I've tested first speed optimised Paq8k2 versions - lp and lp2. Both are about 9% faster than original paq8k2.

Maybe it doesn't look impressive for someone, but any % of paq8k2 speedup is worth mentioning. To test original paq8k2 version on my testbed set of files I've wasted about 100 hours.... and it could be 9 hours less with this version.

Darek

**LovePimple** · 3rd August 2009, 00:10

Thanks for the feedback!

Are my latest experimental builds (download from SendSpace) any faster?

**Darek** · 3rd August 2009, 14:29

I'm still testing.
For raffly lookup: paq8k2_opt4 version shows the best scores - about 1.5% faster than paq8k2_lp. It means 10-11% speedup from original paq8k2.
Darek

**Darek** · 6th August 2009, 11:40

Hi!
some feedback about new compilations. Due to very slow algorithm I've tested only some files.
In average the best compilation is opt3 version which speedup compression about 15%! However some files (textual) could speedup up to about 21%! Exe file have got minor speedup - 12.5%.

Opt3 version wins becouse shows the biggest speedup on the biggest files, but opt4 is faster on smaller files.

In attached xls are detailed scores for 7 my testbed files.

Darek

**LovePimple** · 6th August 2009, 19:12

Thanks for testing!

**Skymmer** · 13th August 2009, 04:50

A couple of modest speed tests for PAQ8k2 -6 on AMD Athlon 64 4000+ (Single core)

Code:

             |    #1   |    #2   |
-------------|---------|---------|
paq8k2_nodll | 623.151 | 570.387 |
paq8k2_lp    | 582.934 | 537.639 |
paq8k2_lp2   | 590.837 | 543.733 |
paq8k2_opt1  | 583.208 | 538.038 |
paq8k2_opt2  | 590.320 | 542.906 |
paq8k2_opt3  | 591.061 | 543.677 |
paq8k2_opt4  | 590.083 | 543.285 |
---------------------------------'

6.45% speedup for #1 and 5.74% for #2. Average speedup is 6.1%.
LP, once again, my compliments

**LovePimple** · 14th September 2009, 22:01

Bill Pettis has released PAQ8K3...

http://ilovemyking.googlepages.com/paq8k3.cpp
http://ilovemyking.googlepages.com/paq8k3.exe

Thanks to Matt Mahoney for the links.

**PiPPoNe92** · 14th September 2009, 23:06

wooow!! it's fast than old versions!!

The compression ratio is abnormal!! I have never seen anything else!

thanks Matt Mahoney and Bill Pettis.

**Jan Ondrus** · 7th February 2010, 19:12

Description:
paq8kx_v4 = paq8kx_v1 + paq8px_v68
paq8kx_v5 = paq8kx_v1 + paq8px_v68p3

**LovePimple** · 7th February 2010, 22:52

Thanks Jan!

Here are my usual builds...

NOTE: _speed_optimised is now renamed _alt (alternative).

**Darek** · 8th February 2010, 13:40

Thanks Jan for new, very powerful versions.
Thanks LovePimple for comilations.

My tests are in attached excel file. To short sum up:

paq8kx_v4 -GREAT version, which breaks almost all my records for files which haven't any special/dedicated compression type in paq algorithm. However - there is one backwardness to old paq8kx_v1 version - all dedicated/specified compressions methods (WAV, GRAPHIC (BMP, TIFF, TGA), JPG, DBF) are (it looks like for me) non-changed to paq8kx_v1. Due to this, compression ratio of files which use this special compression methods is worse than in standard V68 version. Special compression methods for my testbed are statistically the best in mentioned standard V68 version.

paq8kx_v5 - it is noticeably weaker version than v4. In overall is better than V68, however for files which didn't used special compressions, score is worse than paq8px_v1.

Darek

**Bill Pettis** · 8th February 2010, 16:48

I was happy to find this, here's my test:

Notepad2.exe (x64 version)
----------------------

paq8k3 -6: 257k (100%)
pxv68p3 6: 235k (91.4%)
paq8k5 -6: 230k (89.5%)
paq8k4 -6: 229k (89.1%)

Nice work.

**Bill Pettis** · 9th February 2010, 03:25

Mixer update hack, saves a few hundred bytes here.

**Bill Pettis** · 9th February 2010, 07:01

Well, now that "err" is random, I realized that using 2 mixers to do the exact same thing will produce different results. So here it is with 2 identical mixers. It's very slow now, and compresses very slightly better.

**ddfox** · 9th February 2010, 12:25

Originally Posted by Bill Pettis

Well, now that "err" is random, I realized that using 2 mixers to do the exact same thing will produce different results. So here it is with 2 identical mixers. It's very slow now, and compresses very slightly better.

Thanks

**Darek** · 10th February 2010, 11:41

Hi!
There are results of my test of 4a and 4adouble2 versions (in excel file).

To short sum up paq8kx_v4a is an inprovement to version 4 and paq8kx_v4adouble2 shows even more improvement to version 4a. Another record breaking versions! Double2 version is about 2 times slower than version 4, but for bigger files become very, very slow - 3-4 times slower than 4.

Files w/o special compression methods are compressed better than version 4 and even more in comparison vs. v68.
For files with dedicated compression methods (WAV, GRAPHIC (BMP, TIFF, TGA), JPG, DBF) compression is worse than v68 in all cases, and very often worse than previous v4 version.

Darek

**Bill Pettis** · 11th February 2010, 21:09

I also tried doubling the number of "m.set" commands via a loop, but the results were not as well. Apparently there is a need for 2 mixers, as my previous effort. It continues to improve with four mixers, and so on, but I wouldn't release a program with that many mixers...except as a joke.

**Bill Pettis** · 16th February 2010, 17:41

No code, more rambling:

Using 2 identical mixers works great, and differentiating the mixers usually makes things worse. The thing that can change between them is the pseudo-context "order" in ContextModel2. There are many possibilities for how to calculate this--usually by taking the return value from from another model instead of the Normal Model.

**Bill Pettis** · 17th February 2010, 11:58

Originally Posted by Bill Pettis

No code, more rambling:

Using 2 identical mixers works great, and differentiating the mixers usually makes things worse. The thing that can change between them is the pseudo-context "order" in ContextModel2. There are many possibilities for how to calculate this--usually by taking the return value from from another model instead of the Normal Model.

Here's an example of what I just explained.

I think there is a possibility for further differentiation of the mixers, but I am frustrated and going to leave it alone for now. Examine the process used for the determination of order, and change it if you can. I am still assuming that if the two mixers were not identical the compression may improve.

**Jan Ondrus** · 17th February 2010, 14:17

Originally Posted by Bill Pettis

I think there is a possibility for further differentiation of the mixers, but I am frustrated and going to leave it alone for now. Examine the process used for the determination of order, and change it if you can. I am still assuming that if the two mixers were not identical the compression may improve.

Code:

int pr=m2[0].p()+m2[1].p()+1>>1;

What about some more advanced combination of Mixers outputs than simple average?

**Bill Pettis** · 17th February 2010, 15:29

Originally Posted by Jan Ondrus

What about some more advanced combination of Mixers outputs than simple average?

It's possible to make another mixer to combine the outputs of several mixers, I suppose. I tried this a long time ago, but no luck recently.

**Bill Pettis** · 17th February 2010, 21:06

I think this should have better compression than the rest of the kx versions in this thread. It is also a nice speed-up compared to using 2 mixers.

**Darek** · 17th February 2010, 22:38

Hi!
my scores for Paq8kx_v6 are in enclosed file.

For whlole testbed compression ratio is almost identical as for Paq8kx_v4double2 (108 bytes more). Some files are compressed slightly better, some worse than Paq8kx_v4double2, but differences arent higher than some hundreads of bytes - even for bigger files. Compression times are higher than double2 version.

One thing worth to mention - all text files are compressed slightly better than other kx and px versions.

Darek

**Darek** · 18th February 2010, 02:12

And paq8kx_v7 scores - in attached file.

Version 7 has some nice improvements to v6 or 4double2, but only for default compression (not defined files), which cover text files too. Yet another record version!

However, due to some reasons each next version of kx have poorer compression of special files (wave and graphic) than previous. Better default compression causes in worse special compression scores. Due to this effect, for whole test gain from default compression is almost compensated by special files loss and difference between previous and verion 7 for total score is very small.

Darek

**Bill Pettis** · 20th February 2010, 04:04

Originally Posted by Darek

Compression times are higher than double2 version.

That surprised me. But I only ever test very small files. You can't expect me to write code AND test it. :P

**Darek** · 20th February 2010, 13:33

To be honest - I'm only compression ratio freak because compression ratio is the only variable wich is constant over time. I've mentioned compression times approximation to raffly comparison - slower/faster.

Compression time isn't any important factor for me because his scores are only contemporary proper. Of course there are some limitations on extremally slow programs, but in longer time perspective this variable is negligible.

Darek

**Bill Pettis** · 20th February 2010, 17:48

Originally Posted by Darek

To be honest - I'm only compression ratio freak because compression ratio is the only variable wich is constant over time. I've mentioned compression times approximation to raffly comparison - slower/faster.

Compression time isn't any important factor for me because his scores are only contemporary proper. Of course there are some limitations on extremally slow programs, but in longer time perspective this variable is negligible.

Darek

I agree, mostly. I think that as the algorithm improves, the same compression will be attained at faster speeds. I remember PAQ7, when the match model completely took over on a match, causing a huge speed-up but hurting compression later, as the mixer wasn't trained on all those skipped predictions. Somehow there has to be a way to achieve speed and high compression. I'm certainly not looking for it in paq8k, though. Not yet.

**starfruitinc** · 17th March 2010, 08:26

Hello, I'm quite new to coding, but I do know some of the basics. I have a problem with compiling, and the compiler returns errors "undefined reference to 'dot_product'" and "undefined reference to 'train'". Am I missing a file?

Thanks,
SFI

**Intrinsic** · 17th March 2010, 16:24

dot_product, i think that's in the asm file? been a while since i compiled PAQ.
You should be able to compile PAQ without it though, details are in the cpp file as to how.

The filename is paq7asm_sse2.asm and you can find it inside the zip here:
http://encode.dreamhosters.com/showp...&postcount=573

**starfruitinc** · 17th March 2010, 20:14

I tried the file paq7asm_sse.asm and this is the error I have:

Code:

/usr/bin/ld:paq7asm_sse2.asm: file format not recognized; treating as linker script

Then I tried paq7asm.obj, and this is the error I have:

Code:

paq7asm.obj: file not recognized: File format not recognized

I'm not sure what happened.

Thread: PAQ8K

Thread Tools

Search Thread

Display

paq8kx_v4 and paq8kx_v5

Similar Threads

PAQ8K

Tags for this Thread

Posting Permissions