e8c posts from GDC discussion

[e8c]

T2, Rapid.
Modified IZ.
Сам доделать не осилю.

[e8c]

Fix (in attachment).
​Обидно, "уплыл косарь".

[e8c]

​Несколько слов о тайм-лимитах для T2.

Мне нравится, как команда WebP определяет "скоростные режимы": не "fast" и "slow", а "capture mode" и "delivery/storage mode".


Предположим, необходим кодек для покадрового lossless-сжатия видео, захваченного с экрана во время игры. У геймера 8-ядерный проц, из которых только 2 ядра могут быть отданы под real-time сжатие. Видео 1080@60 - это 375 MB/s, то есть минимальная однопоточная производительность кодека = 200 MB/s.


В то же время, победителем в категории "T2, Rapid" может стать асимметричный кодек с соотношением "enc / dec" == "36 s / 4 s" == "28 MB/s / 250 MB/s", который useless для описанного выше варианта применения.

[e8c]

Submission to T2, compiled executable only.

[e8c]

.

[e8c]

Multithreaded GROOT_FORCE, linear scalability.

[Gonzalo]

All of these progs seem overly-tuned to the sample file. They work remarkably well on it but not so much on pretty much any other PNM...

[e8c]

Algorithm Errors as Art:

https://app.box.com/s/gtm698mi8ns8adv62i1s57wgcuj0lab1

[e8c]

Encoder selects predictor using statistical inference.

[e8c]

​https://wiki.termux.com/wiki/Interna...ternal_storage

$ ./build.sh
...
$ ls -l
total 133232
-rwx------ 1 u0_a219 u0_a219       237 Nov 13 21:37 build.sh
-rwx------ 1 u0_a219 u0_a219     18288 Nov 13 21:39 guess_1
-rwx------ 1 u0_a219 u0_a219     18288 Nov 13 21:39 guess_2
-rwx------ 1 u0_a219 u0_a219     18288 Nov 13 21:39 guess_4
-rw------- 1 u0_a219 u0_a219     21018 Nov 13 21:36 guess_with_timer.c
-rwx------ 1 u0_a219 u0_a219       294 Nov 13 21:36 run.sh
drwx------ 2 u0_a219 u0_a219      4096 Nov 12 13:35 storage
-rw------- 1 u0_a219 u0_a219 136323089 Nov 12 14:12 uncompressed.ppm

SM-T820 (APQ8096, 2x Kryo HP 2.15 GHz + 2x Kryo LP 1.6 GHz):

$ ./run.sh
encode,   1 thread :    29 MPx/s
decode,   1 thread :    33 MPx/s
encode,   2 threads:    59 MPx/s
decode,   2 threads:    67 MPx/s
encode,   4 threads:    81 MPx/s
decode,   4 threads:    98 MPx/s

​Планов развивать кодек у меня нет. Вероятно, это последнее сообщение в данной теме. Лицензии на представленный здесь код не будет (и передачи его в "общественную собственность" тоже).

[algorithm]

e8c you are probably the most open person in gdcc . I will probably open source agiannis_image after gdcc.

[e8c]

I will probably open source agiannis_image

Try [to do] it in RFC form with title "Adding a new compression method to PNG": JPEG XL is too slow, WebP 2 limits the image size, both cannot completely replace PNG.

[algorithm]

Indeed e8c. Judging JPEG XL lossless from gdcc results, it is bad. agiannis_image is a very simple codec. It is strange that Alex Rhatushnyak helped in JPEG XL, yet JPEG XL is much worse than QLIC2. I don't understand what's the point of gaining few % while being many times slower.

[Jyrki Alakuijala]

Indeed e8c. Judging JPEG XL lossless from gdcc results, it is bad. agiannis_image is a very simple codec. It is strange that Alex Rhatushnyak helped in JPEG XL, yet JPEG XL is much worse than QLIC2. I don't understand what's the point of gaining few % while being many times slower.

JPEG XL uses decision trees for context modeling. When we get a bit further with the freeze, we can change decision trees to be of a form that is easy to represent as LUTs and speed them up substantially. Also, the predictors of the lossless coding are computed inefficiently at the moment. I'm expecting a ~3x speed up in decoding for lossless.

[e8c]

guess ~ 0.9 MB
Kvick ~ 1.1 MB
agiannis_image ~ 1.7 MB
pglz ~ 2.5 MB

[NCDR]

guess ~ 0.9 MB
Kvick ~ 1.1 MB
agiannis_image ~ 1.7 MB
pglz ~ 2.5 MB

150kb with decode rate ~150mb/s at single core of modern cpu
Guess who?

[Alexander Rhatushnyak]

152868 bytes -- BMF 2.01 -s -q9 -- it is essentially a collection of codecs, with smart detection & switching, per-image afaiu
197737 bytes -- JPEG-XL 0.0.1-6b5144cb -q 100 and the default -s 7
160018 bytes -- same as above, but with -s 9

[e8c]

https://docs.google.com/spreadsheets...#gid=598115372

CLIC: 970'720'188 -133 MB

All other subsets:

Art             604'043'656     -13 MB
Aw              739'756'548     -69 MB
ComixArt        744'836'572     +41 MB
FanArt          558'182'420     -27 MB
Fractals      1'428'366'392     +55 MB
Games         2'445'693'860     -74 MB
ITAP          1'343'411'024     -78 MB
LowPoly         781'223'752      -9 MB
LPCB          1'272'103'344    -128 MB
Manga           640'241'792    +171 MB
PixelArt        166'549'012    +106 MB
Pixiv           509'755'764     -65 MB
SpecArt         886'676'696     -10 MB

​Total: 13'091'561'020 (w/o transparency layer)
Better than 2 entries on the diagram: https://encode.su/threads/3544-JXL-r...ll=1#post67989

[e8c]

​

$ ls -l
​total 404
-rw-rw-r-- 1 abc abc 166924 янв  5 01:08 Vatolin_met_Kadyrov.guess
-rwxrwx--- 1 abc abc 108289 янв  5 01:02 Vatolin_met_Kadyrov.jpg
-rw-rw-r-- 1 abc abc 133429 янв  5 01:07 Vatolin_met_Kadyrov.png

There is only one reason why swapping PNGs for JPGs makes sense: PNG is slow.
I remember that saving 3 MPx screenshots took more than 1 second - it's uncomfortable: instead of interacting with the screenshot, I had to observe a message about incomplete saving the image.
I think that if "guess" becomes part of the PNG, there will be no need to use the lossy format.


Looking at the listing above, it is worth remembering that ".jpg" is source: ".png" and ".guess" store jpg-compression artifacts, without these artifacts compression could probably be better.

[e8c]

​Special processing if tile completely grayscale, and same algorithm as with "-1" if not.

$ ./gray -2 Bennu_Grayscale.ppm Bennu_Grayscale.gray
encode,   2 threads:   118 MPx/s
$ ./gray -d Bennu_Grayscale.gray tmp.ppm
decode,   2 threads:   293 MPx/s
$ ls -l
-rwxrwx--- 1 root vboxsf  101907455 янв 17 00:08 Bennu_Grayscale.emma
-rwxrwx--- 1 root vboxsf  126415072 янв 18 21:37 Bennu_Grayscale.gray
-rwxrwx--- 1 root vboxsf  105285138 янв 16 23:46 Bennu_Grayscale.lea
-rwxrwx--- 1 root vboxsf  128446880 янв 16 19:26 Bennu_Grayscale.png
-rwxrwx--- 1 root vboxsf  732538945 янв 16 19:24 Bennu_Grayscale.ppm

New total for MCIS (Moon Citizen's Image Set) v1.20: 12'745'684'464