Paq8pxd dict

**brispuss** · 18th February 2020, 10:45

I've run some tests with paq8pxd V73 and added the results to the table below.

Tests run under Windows 7 64 bit, with i5-3570k CPU, and 8 GB RAM. Used SSE41 compiles of paq8pxd V*.

Code:

Compressor               Total file(s) size (bytes)    Compression time (seconds)    Compression options
Original 171 jpg files         64,469,752
paq8pxd v69                    51,365,725                          7,753                      -s9
paq8pxd v72                    51,338,132                          7,533                      -s9
paq8pxd v73                    51,311,533                          7,629                      -s9

Tarred jpg files               64,605,696
paq8pxd v69                    50,571,934                          7,897                      -s9
paq8pxd v72                    50,552,930                          7,756                      -s9
paq8pxd v73                    50,530,038                          7,521                      -s9

Overall, improved compression, and slight reduction in compression time for v73!

**Darek** · 18th February 2020, 18:23

Scores on my testset for paq8pxd_v73 - very nice improvements overall - especially for K.WAD file. In total 25KB of gain.
Option -x (second table) gives additional 16KB of gain to -s test -> the gains are visible almost for every file.
Time penalty for -x option compared to -s is about 21%.

**kaitz** · 18th February 2020, 18:48

Originally Posted by brispuss

Overall, improved compression, and slight reduction in compression time for v73!

In single mode there is file overhead about 50 bytes per file vs px version. Like when you compress 1 byte file.
For this single mode test i think its about 9000 bytes total. Not sure how much overhead is on tarred file, probably 100 bytes total for input data. (data that cant be compressed)

**Darek** · 18th February 2020, 18:58

@kaitz - I have question - is there -x option works for -x15 option? I've tested enwik8 files and scores for -s15 and -x15 are identical. Timings are also similar.
For my testset, for textual files there are difference between -s9 and -x9 options.

From other side scores are great:

16'013'638 - enwik8 -s15 by Paq8pxd_v72_AVX2
16'672'036 - enwik8.drt -s15 by Paq8pxd_v72_AVX2

15'993'409 - enwik8 -s15 by Paq8pxd_v73_AVX2
15'956'537 - enwik8.drt -s15 by Paq8pxd_v73_AVX2

**kaitz** · 18th February 2020, 19:03

My bad, level 10 and up will be in -s mode.

**Darek** · 18th February 2020, 20:34

From one side that's good news -> if you could change this then scores for enwik8 could be even better!

**Darek** · 19th February 2020, 11:18

Scores of 4 corpuses for paq8pxd_v73 with -s15 mode. The best scores for Calgary, Canterbury and MaximumCompression for paq8pxd family!

**kaitz** · 19th February 2020, 21:14

paq8pxd_v74

Code:

fix -x option on level 10-15

-x option has affect only on default, text mode.

**Darek** · 19th February 2020, 23:33

@kaitz - at first thanks to fix this. At second "-x option has affect only on default, text mode." - is it ineffective for other types of data?

**kaitz** · 20th February 2020, 01:12

It's used in stream where all default data is. Also for text as humans tend to produce allot of it. :)
Some images have headers and it my gives somewhat better compression on that, but not very useful. It all depends (how many files, etc).

This needs time consuming testing to be actually useful on other types of data.
My test version shows what context are mostly bad for given data in time.

I think there has been good enough improvements from someone like me. But i still wonder.

**Darek** · 20th February 2020, 15:47

some enwik scores for paq8pxd_v74:

16'358'450 - enwik8 -s8 by Paq8pxd_v72_AVX2
16'013'638 - enwik8 -s15 by Paq8pxd_v72_AVX2
16'672'036 - enwik8.drt -s15 by Paq8pxd_v72_AVX2
126'779'432 - enwik9_1423 -s15 by Paq8pxd_v72_AVX2
132'464'891 - enwik9_1423.drt -s15 by Paq8pxd_v72_AVX2

16'339'122 - enwik8 -s8 by Paq8pxd_v74_AVX2
15'993'409 - enwik8 -s15 by Paq8pxd_v74_AVX2
15'956'537 - enwik8.drt -s15 by Paq8pxd_v74_AVX2

16'279'540 - enwik8 -x8 by Paq8pxd_v74_AVX2
15'928'916 - enwik8 -x15 by Paq8pxd_v74_AVX2 - best score for non DRT preprocessed enwik8 file for whole paq8pxd family
15'880'133 - enwik8.drt -x15 by Paq8pxd_v74_AVX2 - best overall score for enwik8 file for whole paq8pxd family - enwik9 score could be even below 126'000'000 bytes

)) I'll check it.

**Darek** · 21st February 2020, 11:11

Scores of 4 Corpuses for paq8pxd_v74 -x15 option. Improvements to paq8pxd_v72 from 0.26% to 0.59% and of course the best scores for all four testsets for paq8pxd series.

**kaitz** · 21st February 2020, 13:36

Originally Posted by Darek

Some enwik scores for paq8pxd_v72:
16'013'638 - enwik8 -s15 by Paq8pxd_v72_AVX2 - tested by Sportman, My compress time 6'811s -@Sportman - you have very fast machine!

On my computer it takes about 9800 seconds to compress.

**CompressMaster** · 21st February 2020, 20:12

Originally Posted by Darek

Scores of 4 Corpuses for paq8pxd_v74 -x15 option. Improvements to paq8pxd_v72 from 0.26% to 0.59% and of course the best scores for all four testsets for paq8pxd series.

Why Silesia and MaximumCompression corpuses aren´t tarred?
Maximum compression corpus - only 40 807 bytes left to get below 6 000 000 bytes! As always, good testing, Darek!

**CompressMaster** · 21st February 2020, 20:15

btw, it´s interesting to see how compression improving over time. And, maybe overestimated expectation, - when did you expect Maximum compression corpus goes below 1 000 000 bytes?

**Darek** · 21st February 2020, 21:48

Originally Posted by CompressMaster

Why Silesia and MaximumCompression corpuses aren´t tarred?
Maximum compression corpus - only 40 807 bytes left to get below 6 000 000 bytes! As always, good testing, Darek!

btw, it´s interesting to see how compression improving over time. And, maybe overestimated expectation, - when did you expect Maximum compression corpus goes below 1 000 000 bytes?

Truely speaking - I don't know. These tests were tested mostly w/o tarred file test from beginnig. I was copy existed approaches.

Maybe due to the fact that there are longer tests and compress additionally tarred file double the test time.
For paq8px/pxd there is a quite reasonable time yet, but for cmix it's additonal 3-day test in one approach which sometimes could be hard to handle.

From other side - tarred file test didn't give you any information about improvement of particular files.

According to MaximumCompression corpus estimate -> 1'000'000 is very ambitious challenge... at now best scores of all files gives 5'872'598 bytes. Using all techniques from all compressors (especially paq8px and paq8pxd for FlashMx.pdf and vcfiu.hlp) into actually best cmix there is a chance to get 5'800'000, maybe 5'700'000 bytes. More parsers and better NN compression maybe gives additonal 100-300KB of gain then we are landing about 5'400'00-5'500'000 bytes. There's need a completely new technique or specialized parsers for all files to get lower score but it could be 4'000'000 bytes. Hmmmm... 1'000'000 looks impossible at now for me.

In attached table there are the best scores of MaximumCompression corpus for most of best compressors.

**kaitz** · 21st February 2020, 23:49

.hlp file has some LZ compressed data, recompress it. And if you target only this set you can gain allot. Jpeg gain 1-2kb gain is possible. On im24 4kb is possible. On dict 3kb is possible But why? And pxd splits data, and sometimes it is bad for compression. Silesia -> samba. Do you care compressor/decompressor size, memory usage, time...
px version has gains on different places, but conisider time/mem/ etc...
In pxd its clear that sometimes adding more models makes it worse. x vs s option. Like adding dmc back to jpeg stream makes it worse... (one present in pxd).
Its hard.

**Darek** · 22nd February 2020, 12:46

enwik9 score for non DRT version and -s15 (next three scores):

126'211'491 - enwik9_1423 -s15 by Paq8pxd_v74_AVX2, time 67'980,66s.
126'599'124 - enwik9_1423.drt -s15 by Paq8pxd_v74_AVX2, time 97604,05s - I don't know why DRT file is compressed 44% more time...
125'752'479 - enwik9_1423 -x15 by Paq8pxd_v74_AVX2, time 91787,59s - record for a paq8pxd series at all!

**kaitz** · 25th February 2020, 18:56

paq8pxd_v75

Code:

- Change wordModel1

enwik8 -s8 is about 16kb smaller. Time should be same, memory usage maybe bit smaller.

**kaitz** · 25th February 2020, 19:37

Originally Posted by Darek

enwik9 score for non DRT version and -s15 (next three scores):

126'211'491 - enwik9_1423 -s15 by Paq8pxd_v74_AVX2, time 67'980,66s.
126'599'124 - enwik9_1423.drt -s15 by Paq8pxd_v74_AVX2, time 97604,05s - I don't know why DRT file is compressed 44% more time...

Its detected as default data and its slower. Compression probably is worse for this reason.

**brispuss** · 26th February 2020, 07:53

I've run some further tests with paq8pxd V75 and added the results to the table below.

Brief tests were done with paq8pxd v74, but there was no improvement in compression with slightly quicker compression times. I didn't think it was worth posting results for paq8pxd v74 tests.

Tests run under Windows 7 64 bit, with i5-3570k CPU, and 8 GB RAM. Used SSE4 compiles of paq8pxd V*.

Code:

Compressor               Total file(s) size (bytes)    Compression time (seconds)    Compression options
Original 171 jpg files         64,469,752
paq8pxd v69                    51,365,725                          7,753                      -s9
paq8pxd v72                    51,338,132                          7,533                      -s9
paq8pxd v73                    51,311,533                          7,629                      -s9
paq8pxd v75                    51,311,427                          7,509                      -s9

Tarred jpg files               64,605,696
paq8pxd v69                    50,571,934                          7,897                      -s9
paq8pxd v72                    50,552,930                          7,756                      -s9
paq8pxd v73                    50,530,038                          7,521                      -s9
paq8pxd v75                    50,528,772                          7,501                      -s9

For v75, improved compression, and slight reduction in compression time!

**Darek** · 26th February 2020, 11:31

Originally Posted by kaitz

Its detected as default data and its slower. Compression probably is worse for this reason.

Quite interesting thing is that enwik8.drt is compressed 0,23% better than pure enwik8 but enwik9.drt is compressed 0,31% worse than pure enwik9 -> that's 0,54% of difference.
Looks like text/word model is more efficient for bigger files than default model. Learning rate is different? Hmmmm....

**Shelwien** · 26th February 2020, 17:57

DRT uses a static dictionary with only ~45k words. I guess enwik9 gets too many unconverted words.

I wonder if nncp preprocessor could be used instead - either directly, or followed by utf16-to-utf8 conversion.

**Darek** · 26th February 2020, 22:32

paq8pxd_v75 scores on my testset with -s9 option. Some better scores mainly for exe and txt files.

**lzhuff** · 27th February 2020, 01:08

Originally Posted by Darek

paq8pxd_v75 scores on my testset with -s9 option. Some better scores mainly for exe and txt files.

@darek, do you still have the paq8pxd48_bwt1 source code ??

**Darek** · 27th February 2020, 12:40

Originally Posted by lzhuff

@darek, do you still have the paq8pxd48_bwt1 source code ??

Here you are only file I have. There is an official source which was published on the forum, however I don't know if it's ok.

**Darek** · 27th February 2020, 12:42

Scores of 4 Corpuses for paq8pxd_v75 -> very nice improvement on Silesia, however there are best scores for all corpuses

.
Additionaly there are -x15 option comparison of v75 vs.v74.

**Darek** · 28th February 2020, 12:31

first enwik scores for paq8pxd_v75:

16'339'122 - enwik8 -s8 by Paq8pxd_v74_AVX2
15'993'409 - enwik8 -s15 by Paq8pxd_v74_AVX2
15'956'537 - enwik8.drt -s15 by Paq8pxd_v74_AVX2

16'279'540 - enwik8 -x8 by Paq8pxd_v74_AVX2
15'928'916 - enwik8 -x15 by Paq8pxd_v74_AVX2
15'880'133 - enwik8.drt -x15 by Paq8pxd_v74_AVX2

16'319'686 - enwik8 -s8 by Paq8pxd_v75_AVX2 - 0.12% of improvement
15'976'838 - enwik8 -s15 by Paq8pxd_v75_AVX2 - 0.10% of improvement
15'934'372 - enwik8.drt -s15 by Paq8pxd_v75_AVX2 - 0.14% of improvement

16'260'265 - enwik8 -x8 by Paq8pxd_v75_AVX2- 0.12% of improvement
15'912'509 - enwik8 -x15 by Paq8pxd_v75_AVX2- 0.10% of improvement -> this score could provide to about 125'58x'xxx bytes for enwik9_1423
15'859'187 - enwik8.drt -x15 by Paq8pxd_v75_AVX2- 0.13% of improvement, best score for paq8pxd series

According to time there are different changes -> from 5-7% up to 18% quicker (for enwik8.drt -x15)!

**Darek** · 2nd March 2020, 13:35

Another scores for enwik:

16'339'122 - enwik8 -s8 by Paq8pxd_v74_AVX2
15'993'409 - enwik8 -s15 by Paq8pxd_v74_AVX2
15'956'537 - enwik8.drt -s15 by Paq8pxd_v74_AVX2
126'211'491 - enwik9_1423 -s15 by Paq8pxd_v74_AVX2
126'599'124 - enwik9_1423.drt -s15 by Paq8pxd_v74_AVX2

16'279'540 - enwik8 -x8 by Paq8pxd_v74_AVX2
15'928'916 - enwik8 -x15 by Paq8pxd_v74_AVX2
15'880'133 - enwik8.drt -x15 by Paq8pxd_v74_AVX2
125'752'479 - enwik9_1423 -x15 by Paq8pxd_v74_AVX2
126'065'722 - enwik9_1423.drt -x15 by Paq8pxd_v74_AVX2

16'319'686 - enwik8 -s8 by Paq8pxd_v75_AVX2 => 0.12% of gain to paq8pxd_v74
15'976'838 - enwik8 -s15 by Paq8pxd_v75_AVX2 => 0.10% of gain to paq8pxd_v74
15'934'372 - enwik8.drt -s15 by Paq8pxd_v75_AVX2 => 0.14% of gain to paq8pxd_v74
126'227'845 - enwik9_1423 -s15 by Paq8pxd_v75_AVX2 => unfortunately, enwik9 got 0.01% lose to paq8pxd_v74 despite enwik8 got nice gain. Time saving is about 3.7%.
126'615'528 estimated - enwik9_1423.drt -s15 by Paq8pxd_v75_AVX2

16'260'265 - enwik8 -x8 by Paq8pxd_v75_AVX2
15'912'509 - enwik8 -x15 by Paq8pxd_v75_AVX2
15'859'187 - enwik8.drt -x15 by Paq8pxd_v75_AVX2

enwik9_1423 -x15 by Paq8pxd_v75_AVX2 is on the run

**Shelwien** · 2nd March 2020, 14:54

@Darek: You can compile it, right?
It could make sense to experiment with ppmd parameters here: https://github.com/kaitz/paq8pxd/blo...pxd.cpp#L12549
o13 m420 (at -s8) seems pretty random, and there're probably better settings for enwik.

Thread: Paq8pxd dict

Thread Tools

Search Thread

Display

Thanks (2):

Thanks:

Thanks:

Thanks:

Thanks (2):

Thanks:

Thanks:

Thanks:

Thanks:

Thanks:

Thanks (4):

Thanks:

Thanks (2):

Thanks:

Thanks:

Thanks:

Similar Threads

FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)

Dict preprocessor

Tags for this Thread

Posting Permissions