Paq8sk

[Gotty]

which m1 context set ?

  m1 ->set(column >> 3 | min(5 + 2 * static_cast<int>(comp == 0), zu + zv),2048);

  m1->set( coef |  min(7, zu + zv),2048);

  m1->set(mcuPos,1024);  

  m1->set( coef |  min(5 + 2 * static_cast<int>(comp == 0), zu + zv),1024);

  m1->set(coef | min(3, ilog2(zu + zv)), 1024);

[suryakandau@yahoo.co.id]

Paq8sk38
- improve jpeg compression
using -8 option on
f.jpg (darek corpus) 112038 bytes -> 81110 bytes
a10.jpg (maximum compression corpus) 842468 bytes -> 623432 bytes
dsc_0001.jpg 3162196 bytes -> 2188750 bytes
there is source code and binary file inside the package.

[Gotty]

Problems are not fixed (except the hashxxk one).
Exe compiled with asserts on (debug mode).

[suryakandau@yahoo.co.id]

Paq8sk39
this is based on paq8pxd90 with
- improve jpeg compression
- improve image24bit compression
using -s8 option on
paq8sk39
f.jpg 112038 bytes -> 80929 bytes
1.bmp 600054 bytes -> 209121 bytes
paq8pxd90
f.jpg 112038 bytes -> 81483 bytes
1.bmp 600054 bytes -> 209726 bytes
​

[suryakandau@yahoo.co.id]

paq8sk39 -s8 a.tif 2870552 bytes -> 358526 bytes
paq8pxd90 -s8 a.tif 2870552 bytes -> 359941 bytes

[Darek]

Comparison of paq8sk38 and paq8sk39 to paq8px v197 and paq8pxd v90 for my testset.
Good improvements for JPG and 24bpp compression - another records for F.JPG, A.TIF, B.TGA.

Sorry, I've forgot to add the table. Now is ok.

[suryakandau@yahoo.co.id]

testing paq8sk39 -s8 on images40(GDCC images open part corpus) the result is
399892087 bytes compressed to 153573411 bytes

[Gotty]

Could you show a list of the results for

paq8pxd90 vs

paq8sk39 (with timings) per file?

(A single number says nothing. Is it better, is it worse? The reader does not know...)

[suryakandau@yahoo.co.id]

​paq8pxd90 -s8 a.tif
Total 2870552 bytes compressed to 359941 bytes.
Time 254.32 sec, used 2031 MB (2130462336 bytes) of memory
paq8sk39 -s8 a.tif
Total 2870552 bytes compressed to 358526 bytes.
Time 604.85 sec, used 2948 MB (3091267934 bytes) of memory

paq8sk39 -s8 f.jpg
Total 112038 bytes compressed to 80929 bytes.
Time 24.81 sec, used 2431 MB (2550024862 bytes) of memory
paq8pxd90 -s8 f.jpg
Total 112038 bytes compressed to 81483 bytes.
Time 9.51 sec, used 1978 MB (2074951232 bytes) of memory

[Gotty]

Thanks!
It looks like it's a huge cost in time and in memory. Compression gain is 0.4% and 0.6% which is very good, but the cost...

[suryakandau@yahoo.co.id]

Paq8sk40
- improve image24bit compression
using -s8 option on darek corpus:
1.bmp
Total 600054 bytes compressed to 208642 bytes.
Time 212.78 sec, used 3338 MB (3500812030 bytes) of memory
a.tif
Total 2870552 bytes compressed to 358705 bytes.
Time 826.07 sec, used 3338 MB (3500812030 bytes) of memory
b.tga
Total 2872818 bytes compressed to 330288 bytes.
Time 914.34 sec, used 3338 MB (3500812030 bytes) of memory

[suryakandau@yahoo.co.id]

Here is source code and binary file of paq8sk40

[suryakandau@yahoo.co.id]

testing it using -s8 option on GDCC images40 open part corpus the result is:
Total 399892087 bytes compressed to 153487870 bytes.
Time 118924.54 sec, used 3338 MB (3500815981 bytes) of memory
here is the binary of paq8sk40

[suryakandau@yahoo.co.id]

testing with paq8sk40 -s8 option on dog.pnm (https://encode.su/threads/3482-e8c-p...GDC-discussion) the result is:
Total 52104185 bytes compressed to 95002 bytes.
Time 13018.44 sec, used 3338 MB (3500812064 bytes) of memory
decompression process is:
Time 12579.00 sec, used 2408 MB (2525141044 bytes) of memory
the output is match with the input

[Gotty]

If by posting these results your intention is to show that the sk version is getting better, then please post the baseline (the results for the previous version). Otherwise we have no clue what the results mean. Is it in improvement? Or is it not?

It's an improvement when the new version compresses better or faster or uses less memory.

For comparison:
paq8px_v198 -8 dog.pnm:
91428 bytes; MEM: 2520 MB; time: 3450 sec

paq8sk40 -s8 dog.pnm
95002 bytes; MEM: 3338 MB; time: 13018 sec

(Time is not comparable as these were running on different machines. I didn't test your baseline (the previous sk version).)

[suryakandau@yahoo.co.id]

Ijust compare sk40 with this thread
https://encode.su/threads/3482-e8c-p...GDC-discussion

[Gotty]

Usually we compress files and do benchmarks to see if there is an improvement over a previous version. Why compare your result with that thread? It says not much. Naturally a paq version should beat most software - it's not fair to compare your results with those.
Is there an improvement over your baseline? This is the question we are interested of.

Here is the result for pax8pxd v90:
paq8pxd_v90 -s8 dog.pnm:
101204 bytes; MEM: 2031 MB, time: 2267 sec

Could you run it for sk39, please?

[suryakandau@yahoo.co.id]

testing with paq8sk39 -s8 option on dog.pnm (https://encode.su/threads/3482-e8c-p...GDC-discussion) the result is:
Total 52104185 bytes compressed to 97875 bytes.
Time 8471.93 sec, used 2948 MB (3091267968 bytes) of memory

[suryakandau@yahoo.co.id]

Paq8sk41
- improve jpeg compression
using -s8
f.jpg (darek corpus)
Total 112038 bytes compressed to 80842 bytes.
Time 17.05 sec, used 1997 MB (2094600089 bytes) of memory
a10.jpg (maximum compression)
Total 842468 bytes compressed to 619930 bytes.
Time 117.97 sec, used 1997 MB (2094600123 bytes) of memory
happy new year 2021 everyone!!!
​

[Gotty]

testing with paq8sk39 -s8 option on dog.pnm (https://encode.su/threads/3482-e8c-p...GDC-discussion) the result is:
Total 52104185 bytes compressed to 97875 bytes.
Time 8471.93 sec, used 2948 MB (3091267968 bytes) of memory

Thank you Surya!
Well done! Compression improvement is significant. Unfortunately time and memory use are also significant.
(I didn't look at the code. I've got no time for code review this time.)

[suryakandau@yahoo.co.id]

Thank you gotty. Happy new year gotty !!

[Sebastianpsankar]

Are you interested in building a revolutionay compression software that can compress gigabytes of data down to kilobytes... I've invented/formulated a revolutionay algorithm... That can change the paradigm of data... Are you interested... I look forward to your response...

[Gotty]

Happy new year, Surya!

[Darek]

Happy new year, Surya!

[suryakandau@yahoo.co.id]

Happy new year darek!!

[suryakandau@yahoo.co.id]

Paq8sk41
- improve jpeg compression
using -s8
f.jpg (darek corpus)
Total 112038 bytes compressed to 80842 bytes.
Time 17.05 sec, used 1997 MB (2094600089 bytes) of memory
a10.jpg (maximum compression)
Total 842468 bytes compressed to 619930 bytes.
Time 117.97 sec, used 1997 MB (2094600123 bytes) of memory
happy new year 2021 everyone!!!
​

Now it can run upto -x15 option. @lucabiondi could you test it on your corpus please ? thank you

[Gotty]

Now it can run upto -x15 option. @lucabiondi could you test it on your corpus please ? thank you

Did you mean the jpeg part of his corpus? Or all of it?

[suryakandau@yahoo.co.id]

only jpeg part on his corpus

[suryakandau@yahoo.co.id]

Paq8sk42
- improve text compression
using -x11 -w option on coronavirus.fasta the result is
Total 1339868341 bytes compressed to 1152683 bytes.
Time 147308.73 sec, used 20428 MB (4241437842 bytes) of memory
​
Maybe the result is smaller and faster than paq8pxv200 with the same level

[suryakandau@yahoo.co.id]

Paq8sk41
- improve jpeg compression
using -s8
f.jpg (darek corpus)
Total 112038 bytes compressed to 80842 bytes.
Time 17.05 sec, used 1997 MB (2094600089 bytes) of memory
a10.jpg (maximum compression)
Total 842468 bytes compressed to 619930 bytes.
Time 117.97 sec, used 1997 MB (2094600123 bytes) of memory
happy new year 2021 everyone!!!
​

Paq8sk43

-change jpegmodel
-fix/change detection (tiff,text)

f.jpg -s8 80751 bytes 33.61 sec
a10.jpg -s8 619637 bytes 209.85 sec
source: https://github.com/skandau/paq8sk


​