Paq8pxd dict

[Darek]

@darek Could you tes paq8sk2 -x11 -w for enwik9 please ?

I could do it. However I've started to test some paq8pxd options for enwik9. My computer have "only" 32GB of ram and testing two instances which overfow the memory making simultaneously tests slower than one nstance at all...
After it I'll test it. I need to test also how long it could take. Compression longer than 2-3 days is a bit risky due to interruption... but I've already tested 2 weeks instances. I'll check.

Questions -> why -x11 option? As far I found I'ts above the HP requiments? Maybe it's would be better to test -x15?

[Shelwien]

Darek: did you try this? https://encode.su/threads/3242-googl...ession-testing

[Darek]

No. Looks promising but this is real?

[moisesmcardona]

No. Looks promising but this is real?

Testing it with my natively compiled paq8px v187 for Linux. It seems to work. You just have to upload the file manually or download with wget. Also, there seems to be a max of 12 hour limit on the virtual machines. Which means it may get shutdown at any time but may run up to 12 hours.

https://research.google.com/colaboratory/faq.html

[Shelwien]

I guess kaitz can add model dump/restore and time limit :)

[schnaader]

I guess kaitz can add model dump/restore and time limit

Also, don't miss Colab Pro - it's not completely free, but $9.99 per month which isn't too expensive and has a 24 hour timeout.

[kaitz]

I have about 2 weeks to test or add new things, then offline until new year.
Reminder, i am regular user so dont expect magical improvments. My target is 3 spot on ltcb.
Still, there are code and math tags in wiki, pdf like processing maybe?. Cant test now as wrt online is broken again. And some models have constant improvments. Maybe testing without them saves testing time.
And if some brave soul adds wit part to cmix and drops some models (paq)....
If you compare to cmix then pxd is on cmix v7 or v8.

[DZgas]

Oh, paq8pxd-stagnation until new year...

Сompress 10000 blank files:

            bytes
tar     5 121 536
zip     1 397 926
7z          4 169
paq8px v136   257
paq8pxd_v11   293
paq8pxd_v25   380
paq8pxd_v40   303
paq8pxd_v50   313
paq8pxd_v55   251
paq8pxd_v58   248
paq8pxd_v67   220 best
paq8pxd_v69   222
paq8pxd_v70   224
paq8pxd_v71   235
paq8pxd_v75   232
paq8pxd_v80   230
paq8pxd_v83   229

[suryakandau@yahoo.co.id]

I could do it. However I've started to test some paq8pxd options for enwik9. My computer have "only" 32GB of ram and testing two instances which overfow the memory making simultaneously tests slower than one nstance at all...
After it I'll test it. I need to test also how long it could take. Compression longer than 2-3 days is a bit risky due to interruption... but I've already tested 2 weeks instances. I'll check.

Questions -> why -x11 option? As far I found I'ts above the HP requiments? Maybe it's would be better to test -x15?

yes its would be better to test using -x15 maybe it result is better or the same as cmix v7 or cmix v8 on enwik9

[Shelwien]

Tried adding a better AC.
Had to use v82s, since there its possible to use SSE prediction for coding directly, without rounding to 12 bits.

              paq8pxd82s paq8pxd82sa // -x7
00s  1,048,576       442       216 // 1M of zero bytes
FFs  1,048,576       443       355 // 1M of FF bytes[*]
dups 1,048,576       557       396 // repeated 100-byte string of random bytes
rand 1,048,576 1,049,202 1,049,198 // 1M chunk of some archive
BOOK1  768,771   182,660   182,646 // calgary book1 [**]
[*] different results for 00s and FFs are caused by paq's 0 turning into 1/32768 and paq's 4095 turning into 1-8/32768
[**] could be 4 bytes less, but flush after header is incompatible with rc tail cutting

[Darek]

enwik9    128613028    -x9 -w    Paq8pxd_v82_AVX2 5162 MB
enwik9    126615900    -x10 -w    Paq8pxd_v82_AVX2 8482 MB
enwik9    126272866    -x10 -w    paq8pxd82
enwik9    126290118    -x10 -w    Paq8pxd_v83_AVX2

How much is reordering vs mod_sse, i dont know. Updated above.
​

enwik8    16154836    -x8 -w    Paq8pxd_v82_AVX2
enwik8    16125181    -x8 -w    Paq8pxd_v83_AVX2

125'622'750 - enwik9 -x11 -w by Paq8pxd_v82_AVX2, time 83'126,17s, mem declared 13'074MB, really used 10'850MB - info based higher value on task manager
125'241'759 - enwik9 -x12 -w by Paq8pxd_v82_AVX2, time 84'156,91s, mem declared 16'114MB, really used 13'830MB - info based higher value on task manager
124'992'597 - enwik9 -x13 -w by Paq8pxd_v82_AVX2, time 83'127,00s, mem declared 22'194MB, really used 19'343MB - info based higher value on task manager
124'741'041 - enwik9 -x14 -w by Paq8pxd_v82_AVX2, time 85'134,18s, mem declared 27'422MB, really used 22'625MB - info based higher value on task manager
124'688'860 - enwik9 -x15 -w by Paq8pxd_v82_AVX2, time 85'118,42s, mem declared 33'310MB, really used 26'000MB - info based higher value on task manager

[Shelwien]

// -x7 -w
16,300,372 enwik8.paq8pxd82s 
16,297,531 enwik8.paq8pxd82sa

[kaitz]

paq8pxd_v84

​    - adjust jpeg,im1 predictor
    - add mod_SSE to all predictors exept jpeg
    - add external dictionary command line option
    - add RC to encoder (paq8pxd82sa)
    - change ppmd mem
    - in wordmodel process math tag (wiki)

How to use external dictionary:

paq8pxd_v84   -s8 -eNum,dictfile infile

where Num is minimum freq for external dictfile

paq8pxd_v84   -s8 -e10,dictfile world95.txt

Most smaller files should use minfq 10 or larger.
Do not use it for nci and etc. Unless you make dict file yourself.

Line ending in external dict must be \n (inc. last line)

For wiki:

paq8pxd_v84   -s8 -w -e10,dictfile enwik7
paq8pxd_v84   -s8 -w -e1,dictfile enwik8

(When external dict) some parameters are hard coded in wrt,
utf8 chars are added to dict no matter what. Minfq for dynamic words is *2 and length >3
If it crashes then don't use -e

With https://github.com/byronknoll/cmix/t...ter/dictionary
enwik8 -x15 -w is about 1568xxxx bytes.


Math tag thingy gives only 1kb improvement on enwik8, on eniwk7 it was about 2kb. So its soso.

[moisesmcardona]

Thanks @kaitz. Are any of these changes worth porting to paq8px?

[Sportman]

enwik8:
15,809,103 bytes, 7,522.931 sec., paq8pxd_v84_avx2 -x15 -w

[Darek]

My testset scores for paq8pxd_v84. Looks like this SSE mod worsk completely different than Shelwien implementation. Or it's something else.
In general score is about 8'400 bytes worse than paq8pxd_v83 version. Especially for audio and bigger, 24bit image files compression get worse than previous version.
Comparison to Shelwien version get 14KB difference -> second table... @Kaitz - maybe you could use SSE in the same way as Shelwien do?

[kaitz]

Sorry about that, forgot to round down prediction.

[kaitz]

paq8pxd_v85

fix rounding error

[Sportman]

enwik8:
15,794,654 bytes, 8,417.468 sec., paq8pxd_v85_avx2 -x15 -w

[Darek]

Two questions about use dictionary file:
1. I cannot download the Byron's dictionary file from github... I don't know why, everytime is downloaded 8.5KB of HTML file. How I can download such dictionary? Could I copy only list of the words to the "dic" file?
2. If the dictonary is used -> is there any communicate about it?

I've found that for my doc files use just -e10 (w/o anything else) gives some additional bytes gain....

I cannot use any dictionary with this version... I think. I've the same scores with dictionary name typed or not....

And scores for my testset for paq8pxd_v85 - 8KB of gain and 3KB better score than paq8pxd_v82_SSE. Still not better than paq8pxd_v75 (20KB less), but it's something in good direction
Very good scores for 24bpp image files = A.TIF - second score ever and B.TGA - the best score ever from all compressors!

[kaitz]

Two questions about use dictionary file:
1. I cannot download the Byron's dictionary file from github... I don't know why, everytime is downloaded 8.5KB of HTML file. How I can download such dictionary? Could I copy only list of the words to the "dic" file?

https://raw.githubusercontent.com/byronknoll/cmix/master/dictionary/english.dic
and save where programm is.

2. If the dictonary is used -> is there any communicate about it?

Only when compressing WRT dict count shows total words.

Example world95.txt

No external dictionary:

paq8pxd_v85_AVX2 -s8  world95.txt
Creating archive world95.txt.paq8pxd85 with 1 file(s)...


File list (21 bytes)
Compressed from 21 to 23 bytes.


1/1  Filename: world95.txt (2988578 bytes)
Block segmentation:
 0           | eoltxt    |   2988578 [0 - 2988577]

 Segment data size: 40 bytes

 TN |Type name |Count      |Total size
-----------------------------------------
 34 |eoltxt    |         1 |   2988578
-----------------------------------------
Total level  0 |         1 |   2988578

text wrt stream(9).  Total 2899483
compressed stream(12).  Total 627
 Total 2899483 wrt: 1830527
WRT dict count 1841 words.
^CTerminate batch job (Y/N)? n

External dictionary:

paq8pxd_v85_AVX2 -s8  -e10,english.dic.txt  world95.txt
Creating archive world95.txt.paq8pxd85 with 1 file(s)...


File list (21 bytes)
Compressed from 21 to 23 bytes.


1/1  Filename: world95.txt (2988578 bytes)
Block segmentation:
 0           | eoltxt    |   2988578 [0 - 2988577]

 Segment data size: 40 bytes

 TN |Type name |Count      |Total size
-----------------------------------------
 34 |eoltxt    |         1 |   2988578
-----------------------------------------
Total level  0 |         1 |   2988578

text wrt stream(9).  Total 2899483
compressed stream(12).  Total 627
 Total 2899483 wrt: 1853647
WRT dict count 2615 words.
^CTerminate batch job (Y/N)? n

Maybe -e11 works better... or -e15, its endless

enwik8    15862215    -s15 -w    Paq8pxd_v85_AVX2
​enwik8    15749003    -s15 -w -e1,english.dic.txt    Paq8pxd_v85_AVX2

[Darek]

@Kaitz - last question for a while - is english.dic file name which you use : "english.dic" or "english.dic.txt" ?

[kaitz]

Yes, its name as any file.

[Darek]

One thing which is strange. I've mentioned it above but I've checked it double and compression and decompression is OK:

paq8pxd_v85 -x9 R.DOC => 27'333 bytes
paq8pxd_v85 -x9 -e1 R.DOC => 27'290 bytes - funny, just use -e1 w/o dictionary gives 43 bytes of gain

paq8pxd_v85 -x9 S.DOC => 24'284 bytes
paq8pxd_v85 -x9 -e1 S.DOC => 24'123 bytes - the same effect - lonely -e1 gives 161 bytes of gain

This effect is visible only with these two files from my testset.

[moisesmcardona]

What does the -e flags do?

[kaitz]

One thing which is strange. I've mentioned it above but I've checked it double and compression and decompression is OK:

paq8pxd_v85 -x9 R.DOC => 27'333 bytes
paq8pxd_v85 -x9 -e1 R.DOC => 27'290 bytes - funny, just use -e1 w/o dictionary gives 43 bytes of gain

paq8pxd_v85 -x9 S.DOC => 24'284 bytes
paq8pxd_v85 -x9 -e1 S.DOC => 24'123 bytes - the same effect - lonely -e1 gives 161 bytes of gain

This effect is visible only with these two files from my testset.

It was not intended but means that dynamic minimum word count is multiple of 2. So probably only some words or none go to dictionary. As stated above, WRT word count shows difference.

What does the -e flags do?

Selects minimum count for word in external dictionary file that must be present in compressed file. Otherwise not used.

[moisesmcardona]

Great work on v85! TIFF file compression is better than on paq8px

[Darek]

enwik scores fr paq8pxd v85 with Byron's dictionary:

15'674'853 - enwik8 -x15 -w -e1,english.dic by Paq8pxd_v85, time 10'775,22s
123'523'709 - enwik9 -x15 -w -e1,english.dic by Paq8pxd_v85, time 114'420,64s

Two best scores for whole enwik8 serie

[kaitz]

enwik scores fr paq8pxd v85 with Byron's dictionary:

15'674'853 - enwik8 -x15 -w -e1,english.dic by Paq8pxd_v85, time 10'775,22s
123'523'709 - enwik9 -x15 -w -e1,english.dic by Paq8pxd_v85, time 114'420,64s

Two best scores for whole enwik8 serie

my enwik8 -x15 was same.

enwik9 124315764 -s15 -w -e1,english.dic.txt Paq8pxd_v85_AVX2

My latest test (-w and wrt changes):

[Darek]

Scores of 4 corpuses for paq8pxd v85. Similar to earlier versions this version also get best scores for all 4 corpuses