Paq8sk

**Darek** · 11th May 2020, 02:06

I've alreadry started paq8sk13 test on enwik9. If it wouldn't be so hard, I'll finish it.
But - @suryakandau - could you use less memory usage to your version?
It's important due to time spend to compression.

Compressor which use less than 32GB like paq8sk7 spend about 31h to compress enwik9 with -x14 option.
Compressor which use more than 32GB like paq8sk13 spend about .... I don't know maybe 55-60h to compress enwik9 with visibe common usage issues with laptop....
There would be only few man to test your compressor. I'm off..

**Sportman** · 11th May 2020, 21:07

enwik8:
15,764,064 bytes, 8,423.297 sec., paq8sk15 -x15 -w
15,636,143 bytes, 8,894.682 sec., paq8sk15 -x15 -w -e1,english.dic

**suryakandau@yahoo.co.id** · 11th May 2020, 22:47

Originally Posted by Sportman

enwik8:
15,764,064 bytes, 8,423.297 sec., paq8sk15 -x15 -w

could you test it using -x15 -w -e1,english.dic ??

**Darek** · 12th May 2020, 12:17

Originally Posted by Sportman

enwik8:
15,755,537 bytes, 11,013.735 bytes, paq8sk14 -x15 -w
15,638,616 bytes, >10,958.687 bytes, paq8sk14 -x15 -w -e1,english.dic

App crash at finish (in time calculation part?).

@Sprortman -> is it should be 11,013.735 seconds and 10,958.687 seconds ?

**Sportman** · 12th May 2020, 16:41

Originally Posted by Darek

@Sprortman -> is it should be 11,013.735 seconds and 10,958.687 seconds _

Oops fixed.

**Darek** · 13th May 2020, 13:31

enwik9 score for paq8sk13:

122'557'871 - enwik9 -x14 -w -e1,english.dic by Paq8sk13, no crash, everything ok and the very good result but:
memory usage = 35'266MB and due to this time was 278909.69 s.... = 3.2 days.

Looking for Sportman scores for sk14 and sk15 versions for enwik8 there would be additional 30KB improvements on enwik9 at all - no need to test it.

**suryakandau@yahoo.co.id** · 13th May 2020, 21:40

Originally Posted by suryakandau@yahoo.co.id

Paq8sk15

the result for dickens file (silesia benchmark) use -s6 -w -e1,english.dic option is
Total 10192446 bytes compressed to 1910568 bytes.Time 1697.33 sec, used 1103 MB (1157402890 bytes) of memory

enwik9 use paq8sk15 -x10 -w -e1,english.dic 124,416,000 bytes use memory about 10.8 gb ram
@darek could u test enwik9 using -x15 -w -e1,english.dic please ?

**Darek** · 13th May 2020, 23:00

This score means that -x15 option could be worse than paq8sk13 about 10-20KB...

Ok. I need to finish some actual tasks and then I'll try. But if this version use more than 30-32GB for -x15 then I'm off.
And as usuall you need to post source code for this version.

**suryakandau@yahoo.co.id** · 17th May 2020, 04:55

Paq8sk18

compress dickens file (silesia benchmark) using paq8sk18 -s6 -w -e1,english.dic
Total 10192446 bytes compressed to 1904185 bytes.
Time 2180.45 sec, used 1238 MB (1299067312 bytes) of memory
the checksum value is match

**Sportman** · 17th May 2020, 14:59

enwik8:
15,758,811 bytes, 8,667.834 sec., paq8sk18 -x15 -w
15,630,141 bytes, 9,069.995 sec., paq8sk18 -x15 -w -e1,english.dic

**suryakandau@yahoo.co.id** · 17th May 2020, 15:37

Paq8sk19

this is the result of dickens file using -x10 -w -e1,english.dic
Total 10192446 bytes compressed to 1884023 bytes.
Time 2261.49 sec, used 15980 MB (3872164038 bytes) of memory

the question is why there is difference with memory written in paq with windows task manager ?
in windows task manager the memory consumed only 11.4-11.5 gb but in paq is 15.9gb ??
i have attached the binary and the source code of this version and g.bat to compile it using mingw 9.2.0

**Shelwien** · 17th May 2020, 15:50

Windows shows actually mapped pages, not simply allocated.
And memory pages are mapped on access.

**Darek** · 17th May 2020, 16:44

Originally Posted by suryakandau@yahoo.co.id

this is the result of dickens file using -x10 -w -e1,english.dic
Total 10192446 bytes compressed to 1884023 bytes.
Time 2261.49 sec, used 15980 MB (3872164038 bytes) of memory

If you compress this file (dickens) with -x10 -w -e10,english.dic then you will got another 5KB of gain. I'm on the test of 4 corpuses with different usage of -e option - if I will found better option then I'll post it.

Memory usage of 15'9GB for -x10 is massive = memory used in -x12 option on paq8pxd.
Probably -x14 option uses much more than 32GB....

**Sportman** · 17th May 2020, 20:31

enwik8:
15,758,738 bytes, 8,639.996 sec., paq8sk19 -x15 -w
15,629,126 bytes, 9,082.471 sec., paq8sk19 -x15 -w -e1,english.dic

**suryakandau@yahoo.co.id** · 17th May 2020, 21:41

Originally Posted by Sportman

enwik8:
15,758,738 bytes, 8,639.996 sec., paq8sk19 -x15 -w

@sportman how much memory by using -x15 option ??

**suryakandau@yahoo.co.id** · 18th May 2020, 00:57

Originally Posted by Shelwien

Windows shows actually mapped pages, not simply allocated.
And memory pages are mapped on access.

so which one we use ? windows task manager or memory written in paq ??

**Sportman** · 18th May 2020, 02:12

Originally Posted by suryakandau@yahoo.co.id

@sportman how much memory by using -x15 option ??

enwik8:
15,786,932 bytes, 8,935.927 sec., paq8sk19 -x15

RamMap show paq8sk19 (-x15) process use 27,850,289,152 bytes total.‬

https://docs.microsoft.com/en-us/sys...wnloads/rammap

**suryakandau@yahoo.co.id** · 18th May 2020, 14:03

Originally Posted by Sportman

enwik8:
15,786,932 bytes, 8,935.927 sec., paq8sk19 -x15

RamMap show paq8sk19 (-x15) process use 27,850,289,152 bytes total.‬

https://docs.microsoft.com/en-us/sys...wnloads/rammap

What about paq8sk19 -x15 -w -e1,English.dic
How much memory it needs ?

**Sportman** · 18th May 2020, 17:38

Originally Posted by suryakandau@yahoo.co.id

What about paq8sk19 -x15 -w -e1,English.dic
How much memory it needs ?

Looks like similar at 41% total 27,324,891,136‬ bytes, -x15 memory sample was at 95%.

**suryakandau@yahoo.co.id** · 21st May 2020, 01:16

Originally Posted by Sportman

enwik8:
15,758,738 bytes, 8,639.996 sec., paq8sk19 -x15 -w
15,629,126 bytes, 9,082.471 sec., paq8sk19 -x15 -w -e1,english.dic

enwik9:
123910093 bytes 251888.76 sec paq8sk19 -x10 -w -e1,english.dic

**Darek** · 21st May 2020, 18:20

Originally Posted by suryakandau@yahoo.co.id

enwik9:
123910093 bytes 251888.76 sec paq8sk19 -x10 -w -e1,english.dic

It looks suspicious - are you shure that the program finish OK? There no any crash at the end?

**suryakandau@yahoo.co.id** · 21st May 2020, 18:33

Originally Posted by Darek

It looks suspicious - are you shure that the program finish OK? There no any crash at the end?

everything is ok

**Darek** · 22nd May 2020, 16:52

Originally Posted by suryakandau@yahoo.co.id

Paq8sk19

this is the result of dickens file using -x10 -w -e1,english.dic
Total 10192446 bytes compressed to 1884023 bytes.
Time 2261.49 sec, used 15980 MB (3872164038 bytes) of memory

the question is why there is difference with memory written in paq with windows task manager ?
in windows task manager the memory consumed only 11.4-11.5 gb but in paq is 15.9gb ??
i have attached the binary and the source code of this version and g.bat to compile it using mingw 9.2.0

Best option for dickens file I can found for dictionary split is -e77,english.dic - this gives about 14.5 KB of gain compared to version w/o dictionary.

**suryakandau@yahoo.co.id** · 27th May 2020, 10:34

Originally Posted by suryakandau@yahoo.co.id

enwik9:
123910093 bytes 251888.76 sec paq8sk19 -x10 -w -e1,english.dic

@darek could you test paq8sk19 -x15 -w -e1,english.dic on enwik9 please ? thank you

**suryakandau@yahoo.co.id** · 29th May 2020, 16:37

Paq8sk22
- improve text model

**suryakandau@yahoo.co.id** · 29th May 2020, 20:54

Originally Posted by suryakandau@yahoo.co.id

Paq8sk22
- improve text model

The result using paq8sk22 -s6 -w -e1,English.dic on Dickens file is 1900420 bytes

**Sportman** · 30th May 2020, 02:47

enwik8:
15,755,063 bytes, 14,222.427 sec., paq8sk22 -x15 -w
15,620,894 bytes, 14,940.285 sec., paq8sk22 -x15 -w -e1,english.dic

**suryakandau@yahoo.co.id** · 30th May 2020, 07:21

Originally Posted by Sportman

enwik8:
15,755,063 bytes, 14,222.427 sec., paq8sk22 -x15 -w

How about paq8sk22 -x15 -w -e1,english.dic for enwik8

**Darek** · 30th May 2020, 15:29

Originally Posted by Sportman

enwik8:
15,755,063 bytes, 14,222.427 sec., paq8sk22 -x15 -w
15,620,894 bytes, 14,940.285 sec., paq8sk22 -x15 -w -e1,english.dic

@Sportman - it's dramatic change in compression time - does this version use much more memory than previous?

**Darek** · 30th May 2020, 15:33

Originally Posted by suryakandau@yahoo.co.id

@darek could you test paq8sk19 -x15 -w -e1,english.dic on enwik9 please ? thank you

I will. At least I'll try

I need 2-3 days to finish task which is in progress and then I'll start paq8sk19.

paq8sk22 looks for me like move in not good direction - very slightly improvrment affected double of compression time.

Thread: Paq8sk

Thread Tools

Search Thread

Display

Thanks (2):

Thanks:

Thanks:

Thanks:

Thanks (2):

Thanks (2):

Thanks:

Thanks (3):

Thanks:

Thanks:

Thanks (2):

Posting Permissions