Paq8sk

[suryakandau@yahoo.co.id]

enwik8:
15,774,291, 8,947.975 sec., paq8sk9 -x15 -w

@darek/sportman could you test it using -x15 -w -e1,english.dic please ?

[Sportman]

enwik8:
15,656,780 bytes, 9,351.442 sec., paq8sk9 -x15 -w -e1,english.dic

[Darek]

@Sportman - will you try to compress enwik9 or I got this?

[xinix]

Sportman, Darek
This is just a complete clone Paq8pxd.
Why test it?

[Sportman]

will you try to compress enwik9?

Only enwik8.

[Sportman]

This is just a complete clone Paq8pxd. Why test it?

It's improved and generate smaller output files.

I don't mind people copying/cloning open source or reverse enginering parts of none commercial software as long it do not violate the supplied license/copyright (if any) and credits are given or permission is asked and given (if alive).

[kampaster]

It's improved and generate smaller output files.

I don't mind people copying/cloning open source or reverse enginering parts of none commercial software as long it do not violate the supplied license/copyright (if any) and credits are given or permission is asked and given (if alive).

Something I do not see the source code from April 14 to May 4.
___
This is a waste of time and your resources. We haven't tested the original pxd enough.

[Darek]

Something I do not see the source code from April 14 to May 4.
___
This is a waste of time and your resources. We haven't tested the original pxd enough.

That's good point...

@Suryakandau - could you post also sources for latest versions as is required for use this code?

[suryakandau@yahoo.co.id]

Something I do not see the source code from April 14 to May 4.
___
This is a waste of time and your resources. We haven't tested the original pxd enough.

I will post the source code later. Thanx

[Darek]

Ok, we wait for the source then.
Despite this you also need to back to pre v84 (paq8pxd) version builds changes:
https://encode.su/threads/1464-Paq8p...ll=1#post64855

[suryakandau@yahoo.co.id]

Paq8sk13

this is source and binary of paq8sk13. the result for dickens file(silesia benchmark) is

Total 10192446 bytes compressed to 1901677 bytes.
Time 2230.69 sec, used 1341 MB (1406994532 bytes) of memory

the decompression time is Time 2230.34 sec, used 1192 MB (1250618671 bytes) of memory
the checksum value is match

[suryakandau@yahoo.co.id]

Paq8sk13

this is source and binary of paq8sk13. the result for dickens file(silesia benchmark) is

Total 10192446 bytes compressed to 1901677 bytes.
Time 2230.69 sec, used 1341 MB (1406994532 bytes) of memory

the decompression time is Time 2230.34 sec, used 1192 MB (1250618671 bytes) of memory
the checksum value is match

I use -s6 -w -e1,english.dic to compress
To decompres use -d -e1,English.dic

[suryakandau@yahoo.co.id]

Paq8sk9


NOw this version using external dictionary, so the result for enwik9 using -x10 -w -e1,english.dic
Total 1000000000 bytes compressed to 124758108 bytes.
Time 227725.05 sec, used 11672 MB (3649474565 bytes) of memory


paq8sk10 is running now for enwik9

Paq8sk10

the enwik9 result using paq8sk10 -x10 -w -e1,english.dic is
Total 1000000000 bytes compressed to 124670776 bytes.
Time 227228.65 sec, used 14107 MB (1907427393 bytes) of memory

[Sportman]

enwik8:
15,756,129 bytes, 11,220.447 sec., paq8sk13 -x15 -w
15,639,539 bytes, >11,010.312 sec., paq8sk13 -x15 -w -e1,english.dic

App crash at finish.

[suryakandau@yahoo.co.id]

enwik8:
15,756,129 bytes, 11,220.447 sec., paq8sk13 -x15 -w

App crash at finish.

Just use -x14

[Darek]

What a difference between v10 and v13?
Which is more advanced?

[suryakandau@yahoo.co.id]

What a difference between v10 and v13?
Which is more advanced?

More advanced is v13

[suryakandau@yahoo.co.id]

What a difference between v10 and v13?
Which is more advanced?

Could you test paq8sk13 for enwik9 use -x14 -w -e1,English.dic please ? Thank you

[suryakandau@yahoo.co.id]

enwik8:
15,756,129 bytes, 11,220.447 sec., paq8sk13 -x15 -w
15,639,539 bytes, >11,010.312 sec., paq8sk13 -x15 -w -e1,english.dic

App crash at finish.

I guess it is caused by memory requirement, bigger than 32 GB for -x15. If use -x14 it is ok

[suryakandau@yahoo.co.id]

Paq8sk13

this is source and binary of paq8sk13. the result for dickens file(silesia benchmark) is

Total 10192446 bytes compressed to 1901677 bytes.
Time 2230.69 sec, used 1341 MB (1406994532 bytes) of memory

the decompression time is Time 2230.34 sec, used 1192 MB (1250618671 bytes) of memory
the checksum value is match

the enwik9 result is
Total 1000000000 bytes compressed to 124406262 bytes.
Time 235686.44 sec, used 13882 MB (1672175516 bytes) of memory

[suryakandau@yahoo.co.id]

the enwik9 result is
Total 1000000000 bytes compressed to 124406262 bytes.
Time 235686.44 sec, used 13882 MB (1672175516 bytes) of memory

Maybe the result of use -x15 can be 120.xxx.xxx bytes but it needs > 32 GB ram

[Darek]

the enwik9 result is
Total 1000000000 bytes compressed to 124406262 bytes.
Time 235686.44 sec, used 13882 MB (1672175516 bytes) of memory

According to memory usage you did use -x10 option, right?
Using memory above 32GB is most probably over my possibilities.

[suryakandau@yahoo.co.id]

According to memory usage you did use -x10 option, right?
Using memory above 32GB is most probably over my possibilities.

maybe you can test using -x14. thank you

[Darek]

I've started to test -x14 option, according to Sportman's information that -x15 options generate crash.
My estimate of paq8sk13 with-x14 option for enwik9 is about 122'5xx'xxx to 122'6xx'xxx. Time to compress on my laptiop = 36h (30h to go). Option -x15 isn't much better. Maybe 50KB, maybe 100KB less.

One question more - is the paq8pxd1.cpp file a source of paq8sk13?

[suryakandau@yahoo.co.id]

I've started to test -x14 option, according to Sportman's information that -x15 options generate crash.
My estimate of paq8sk13 with-x14 option for enwik9 is about 122'5xx'xxx to 122'6xx'xxx. Time to compress on my laptiop = 36h (30h to go). Option -x15 isn't much better. Maybe 50KB, maybe 100KB less.

One question more - is the paq8pxd1.cpp file a source of paq8sk13?

​yes

[Darek]

Hmmm, -x14 option probably use much more memory than 32GB... compression start to huge slowdown due to use big swap file. I'm not sure if the process could go to the end. And it's "only" -x14 option then -x15 could require even much memory.

@suryakandau - could you limit next version memory usage to 32GB for -x15?

Maybe that's a reason of crashes - too much memory usage. From other side - crunching numbers by exdend memory usage is not the way we should go....

[suryakandau@yahoo.co.id]

Paq8sk14

here is the latest version of paq8sk with a little improvement on hash function and reduced memory usage.@sportman/darek could you try it use -x15 or -x14 please ?

[Sportman]

enwik8:
15,755,537 bytes, 11,013.735 sec., paq8sk14 -x15 -w
15,638,616 bytes, >10,958.687 sec., paq8sk14 -x15 -w -e1,english.dic

App crash at finish (in time calculation part?).

[suryakandau@yahoo.co.id]

Paq8sk15


the result for dickens file (silesia benchmark) use -s6 -w -e1,english.dic option is
​Total 10192446 bytes compressed to 1910568 bytes.Time 1697.33 sec, used 1103 MB (1157402890 bytes) of memory

[suryakandau@yahoo.co.id]

enwik8:
15,755,537 bytes, 11,013.735 bytes, paq8sk14 -x15 -w

App crash at finish (in time calculation part?).

i guess it comes from memory usage, use -x14 it is ok