ICIP Grand Challenge on Image Compression - Results

[thorfdbg]

Hi folks,

ICIP was last month, and as advertised earlier on this forum, we run a challenge on lossy and lossless image compression there. Amongst the lossy candidates we had JPEG, JPEG 2000, JPEG XR, HEVC, Daala and WebP in the test plan. For lossless, we had many variants of JPEG (including hierarchical), JPEG (predictive), JPEG XR, JPEG 2000, JPEG LS, PNG and FLIF.

By resolution of the committee, we are happy to make the ICIP presentation publicly available here:

https://jpeg.org/downloads/aic/wg1n7..._challenge.pdf

Some reviews:

For lossy, HEVC is the winner, followed closely (and sometimes even left behind by) Daala, which performs very well on face images. Daala also improved from the previous test on the bike test due to smarter filtering.

Surprisingly, "JPEG on Steroids" can even compete with these methods for higher bitrates. It is a JPEG variant with uses some advanced coding tricks similar to mozjpeg. Luckely, I got the permission from the owner to make it available on my github pages under GPLv3, I just have to do it. It will appear there in a couple of days, I will post again as soon as I uploaded it.

WebP is not really any better than existing technology.

JPEG 2000 was unfortunately misconfigured due to an oversight on my side - I run it with the 5/3 wavelet rather than the 9/7 wavelet. My fault. Sorry about that, so please disregard its graphs for now.

Worst performing is JPEG. So what can we learn? It depends a lot on what precisely you do.

For lossless, the winner is FLIF. However, the distance to JPEG-LS part-2 is not very big. FLIF is rather high-complexity, JPEG-LS 2 a low-complexity codec. I guess it's just the usual with lossless compression: If you try something simple, you get 2:1. If you try very hard, you get 2:1. Or something that is a little bit better. After all, you are mostly compressing noise.

I want to thank all participants for submitting their software, and for the test labs for running the subjective tests.

Greetings,

Thomas

[Jyrki Alakuijala]

Which objective measurement correlates most with the subjective measurements (for example by spearman rank)?

Would it be possible to include butteraugli in the objective measurements and correlate it against the subjective measurements, too?

It seems to me that the highest quality (1-1.5 bits per pixel) subjective tests are still in the working range of butteraugli.

[thorfdbg]

Which objective measurement correlates most with the subjective measurements (for example by spearman rank)?

Unfortunately, I'm one of the few participants without a vision lab, so I cannot tell, and I do not have the raw scores. But if you are interested, please drop me a private mail and I'll get you in touch with Touradj. He should have this data. Actually, also from Switzerland, EPFL.

[Intrinsic]

I can't access that PDF atm? 404 not found.

[thorfdbg]

I can't access that PDF atm? 404 not found.

Might be because the ISO livelink system is on maintenance until Monday. So it will be back (hopefully) after the weekend. Unfortunately, I don't have a copy, I could also only pull it out of livelink - which is right now unfortunately down. Update: It seems that Tim has moved the document, so the new link is here: https://jpeg.org/downloads/aic/wg1n7..._challenge.pdf Sorry for the confusion. You also get it by first clicking on "News" on www.jpeg.org, there go to the Chengdu meeting, then scroll down until you reach the section on the grand challenge, and there you find the link.

[skal]

Hi,

WebP is not really any better than existing technology.

well, fwiw, i consider it a satisfying result that it consistently ranked ~3rd, including for lossless, given the constraints.

my .02c
skal/

[Jarek]

skal, WebP was originally a single frame from VP8 - do you plan to switch to a single frame from VP10/AV1?

[skal]

Jarek,

skal, WebP was originally a single frame from VP8 - do you plan to switch to a single frame from VP10/AV1?

As it is right now, from tests, VP10/AV1 could give a 20-30% size gain over WebP, depending on how you count. So, yes, it's definitely in the 'worth it' zone to have a look at next-gen compression techniques. How and when to deploy it is a separate issue.
Re-using a video-codec as-is ( like was done for VP8 ) is asking for troubles, as you can't really fit both requirements for video and images simultaneously. Video want smaller dimension than images, but can afford more memory. Images could use larger dimension but memory is constrained and parallelism should be at pixel level, not frame level. etc. etc.
Lastly, note that some Daala techniques are being retrofitted into AV1 already.

[Alexander Rhatushnyak]

It seems that Tim has moved the document, so the new link is here: https://jpeg.org/downloads/aic/wg1n7..._challenge.pdf Sorry for the confusion.

May I host this document on LPCB site?

[Jon Sneyers]

For lossless, the winner is FLIF. However, the distance to JPEG-LS part-2 is not very big. FLIF is rather high-complexity, JPEG-LS 2 a low-complexity codec. I guess it's just the usual with lossless compression: If you try something simple, you get 2:1. If you try very hard, you get 2:1. Or something that is a little bit better. After all, you are mostly compressing noise.

That is true for photographs (or in general, for any image that is the result of noisy sensors), but not for other kinds of images which do not contain noise. E.g. rendered images, screenshots, icons, logos, plots, webcomics, pixel art, etc. Those weren't tested at the ICIP'16 challenge, but for those type of images, lossless compression can usually achieve much better results, often even producing smaller files than lossy methods. And the gap between FLIF and other lossless formats tends to be larger for such images too.

[Embee]

Very interesting to see some recent measurements lined up. I'm particularly interested in the power efficiency (simple put: the time it takes) of lossy algorithms, any place to find information on that? It's not in the pdf.

[thorfdbg]

Very interesting to see some recent measurements lined up. I'm particularly interested in the power efficiency (simple put: the time it takes) of lossy algorithms, any place to find information on that? It's not in the pdf.

We didn't look into this problem at this time, and it is much more a matter of the implementation. I did something in this direction a couple of years ago in an SPIE publication where I looked at computation time over compression factor. Unsurprisingly, JPEG 2000 was much more dependent on the compression factor compared to JPEG, but for high compression rates, JPEG 2000 was actually better in terms of quality/complexity compared to JPEG. If there is interest, I can probably dig this paper out.

[Tamar]

Isn't HEVC a video compression codec? I'm pretty confused what is the meaning of using this codec as image compressor...
Can you shade light abuot this?
Thanks!

Hi folks,

ICIP was last month, and as advertised earlier on this forum, we run a challenge on lossy and lossless image compression there. Amongst the lossy candidates we had JPEG, JPEG 2000, JPEG XR, HEVC, Daala and WebP in the test plan. For lossless, we had many variants of JPEG (including hierarchical), JPEG (predictive), JPEG XR, JPEG 2000, JPEG LS, PNG and FLIF.

By resolution of the committee, we are happy to make the ICIP presentation publicly available here:

https://jpeg.org/downloads/aic/wg1n7..._challenge.pdf

Some reviews:

For lossy, HEVC is the winner, followed closely (and sometimes even left behind by) Daala, which performs very well on face images. Daala also improved from the previous test on the bike test due to smarter filtering.

Surprisingly, "JPEG on Steroids" can even compete with these methods for higher bitrates. It is a JPEG variant with uses some advanced coding tricks similar to mozjpeg. Luckely, I got the permission from the owner to make it available on my github pages under GPLv3, I just have to do it. It will appear there in a couple of days, I will post again as soon as I uploaded it.

WebP is not really any better than existing technology.

JPEG 2000 was unfortunately misconfigured due to an oversight on my side - I run it with the 5/3 wavelet rather than the 9/7 wavelet. My fault. Sorry about that, so please disregard its graphs for now.

Worst performing is JPEG. So what can we learn? It depends a lot on what precisely you do.

For lossless, the winner is FLIF. However, the distance to JPEG-LS part-2 is not very big. FLIF is rather high-complexity, JPEG-LS 2 a low-complexity codec. I guess it's just the usual with lossless compression: If you try something simple, you get 2:1. If you try very hard, you get 2:1. Or something that is a little bit better. After all, you are mostly compressing noise.

I want to thank all participants for submitting their software, and for the test labs for running the subjective tests.

Greetings,

Thomas

[quikee]

Isn't HEVC a video compression codec? I'm pretty confused what is the meaning of using this codec as image compressor...

Should a video codecs be able to compress a video with only 1 frame? If yes - there you go... that's essentially the same as an image.

Essentially all video codecs are image codecs with additional logic to exploit the similarities in future and/or past frames/images, however specifically made image codecs can be more efficient as the same logic can hold the image codec down (complexity vise).