Diverse third-party ecosystem for optimization of webp and JPEG-XR images

[SolidComp]

Hi all,

As you're well aware, there are lots of JPEG and PNG optimizers and encoders, both lossy and lossless. However, newer image formats like webp and JPEG-XR each have only one reference encoder, from Google and Microsoft, respectively, and no GUI tools for working with these formats.

Do the webp and JPEG-XR formats have the sort of headroom for optimization that JPEG and PNG do? Is there anything about the formats that has prevented the emergence of third-party encoders and optimizers? Have any of you looked at building new encoders or optimizers?

Cheers,

Joe

[Jyrki Alakuijala]

Hi all,

As you're well aware, there are lots of JPEG and PNG optimizers and encoders, both lossy and lossless. However, newer image formats like webp and JPEG-XR each have only one reference encoder, from Google and Microsoft, respectively, and no GUI tools for working with these formats.

Do the webp and JPEG-XR formats have the sort of headroom for optimization that JPEG and PNG do? Is there anything about the formats that has prevented the emergence of third-party encoders and optimizers? Have any of you looked at building new encoders or optimizers?

WebP engineering team responds to issues and proposals at: https://bugs.chromium.org/p/webp/issues/list -- so it is possible to ask them directly why they don't use a guetzli like approach.

About why we build Guetzli and ZopfliPNG:

We built guetzli on top of jpeg, because jpeg performed better at high quality than other alternatives, particularly the lack of a good integral transform and the YUV420 color space reduce the performance of WebP at highest quality (break even with libjpeg was around quality 77, and worse above 85 in libjpeg quality). Better meaning here: faster decode (up to around 6x faster) and less bytes needed at high quality (in comparison to butteraugli scores). New codecs are often far superior at very low bit rates, but that was not what we were interested of. At very low bit rates the images will always be distorted in various ways, typically they are made smoother and the observable surface properties are replaced by plastic feel.

We built ZopfliPNG just because we could -- after building Zopfli and Lode being the author of LodePNG, it was a natural thing to build ZopfliPNG, too. No big strategic thinking. (And naturally ZopfliPNG is a worse solution than our previous work with the WebP lossless format. Also, ZopfliPNG is worse than sending an uncompressed but filtered PNG with brotli content encoding.)

[thorfdbg]

Do the webp and JPEG-XR formats have the sort of headroom for optimization that JPEG and PNG do? Is there anything about the formats that has prevented the emergence of third-party encoders and optimizers? Have any of you looked at building new encoders or optimizers?

Actually, the JPEG XR reference software is *not* from Microsoft. Microsoft funded its development. The HDPhoto Developer Kit is from Microsoft, but this is a different software, and it is neither exactly JPEG XR. The differences are subtle, though, and mostly address defects we found during standardization. As for JPEG XR: Yes, there is a little bit of headroom. It offers variable quantization which is not used by the reference software by default. This alone can give you a PSNR advantage of 0.2dB (depending on the image, if I recall correctly, but I can dig out the publication). Also, there is *some* possibility to tune it towards other quality indices like SSIM - though it is clearly not as flexible (nor as good) as JPEG 2000 is.

[SolidComp]

Actually, the JPEG XR reference software is *not* from Microsoft. Microsoft funded its development. The HDPhoto Developer Kit is from Microsoft, but this is a different software, and it is neither exactly JPEG XR. The differences are subtle, though, and mostly address defects we found during standardization. As for JPEG XR: Yes, there is a little bit of headroom. It offers variable quantization which is not used by the reference software by default. This alone can give you a PSNR advantage of 0.2dB (depending on the image, if I recall correctly, but I can dig out the publication). Also, there is *some* possibility to tune it towards other quality indices like SSIM - though it is clearly not as flexible (nor as good) as JPEG 2000 is.

Thanks for the info. This is interesting. For some reason I thought JPEG-XR was supposed to be better than JPEG 2000 – I think I saw a research paper to that effect. JPEG-XR will probably die off soon since Microsoft has been so lazy with it, and no one else is exploiting its headroom, HDR and other features, etc. or building better encoders, e.g. a libjpegxr-turbo or MozJPEG-XR or a Guetzli approach.

If JPEG 2000 is better, why don't people use it? It's always seemed like a confusing and difficult format to adopt. Just viewing JPEG 2000 images on Windows 10 is difficult. I had to do some hacking to help my girlfriend view her MRIs in some format called DICOM where some of the images were JPEG 2000. Is it patent encumbered? I remember hearing something about that.

Ancillary question: Do you happen to know if Safari on macOS and iOS actually displays JPEG 2000 files? It's supposed to, but I've never been able to find any mention of it in Apple's documentation, and I have no devices to test. That also makes me wonder if patent issues have prompted Apple to keep quiet about it.

[Jyrki Alakuijala]

If JPEG 2000 is better, why don't people use it?

Better is a complicated word.

JPEG 2000 is better than JPEG at very low bit rates for both human viewers and for simple metrics like PSNR.

JPEG 2000 is better (10 % less bloaty) than JPEG at high quality for simple metrics like PSNR.

JPEG 2000 is more feature-rich.

JPEG 2000 is worse (10 % more bloaty) than JPEG at high quality for human viewers.

Most of the digital images overall -- and particularly the images in the internet are high quality images consumed by human viewers. JPEG fits this purpose better than JPEG 2000.

[SolidComp]

Better is a complicated word.

JPEG 2000 is better than JPEG at very low bit rates for both human viewers and for simple metrics like PSNR.

JPEG 2000 is better (10 % less bloaty) than JPEG at high quality for simple metrics like PSNR.

JPEG 2000 is more feature-rich.

JPEG 2000 is worse (10 % more bloaty) than JPEG at high quality for human viewers.

Most of the digital images overall -- and particularly the images in the internet are high quality images consumed by human viewers. JPEG fits this purpose better than JPEG 2000.

I meant better than JPEG-XR, which is what thorfbdg had said. Of course JPEG-2000 is better than JPEG – I mean it has 2000 in the name That's what people did in the 1990s when they wanted to mark something as futuristic and high-tech.

I also thought that JPEG-XR came after JPEG-2000, which implied to me that it would have to be an improvement over it. And it wasn't as messy on the spec side – have you seen all the JPEG-2000 specs?

[SolidComp]

By the way, this reminds me that the plethora of available tools to optimize JPEGs and PNGs makes the case for webp (and JPEG-XR) weaker, or at least more complicated. In reality, webp is not competing with libjpeg or even libjpeg-turbo. webp is competing with the gauntlet of tools that I can put JPEGs through, including lossy optimizers. Think of everything that's in FileOptimizer, plus TinyPNG.com (which does JPEGs too), etc. The same applies to PNGs.

Google must have beautiful, rich, and useful data on webp performance in all aspects, but for some reason it refuses to release it. All I've seen are old comparisons between webp and PNGs where they used pngcrush and pngout, for example. But in the wild we're not limited to pngcrush and pngout, and those aren't even the most powerful tools. Does webp actually produce smaller files than JPEG and PNG if we use all the tools available for those legacy formats? I don't know. I'm not sure if anyone knows. My hunch is that it probably does, but the difference isn't as big as Google claims in its dated tests on the webp website. I'm more confident that webp can beat superoptimized PNGs than superoptimized JPEGs, but the JPEG case is the most complicated because of the difficulty of apples-to-apples comparison in terms of "quality". We have quality integers that we're throwing around that aren't commensurable across formats and most web development shops don't have enough staff to deep dive into the finer points of image optimization.

[SolidComp]

I've also wondered about combining raster and vector approaches in the same format, since it's clear that some images would benefit much more from one or the other.

I also want to see a format that explicitly accounts for image sets in addition to standalone images, since a lot of groups of images on webpages, social media, etc. feature the same settings and subjects and have lots of encodable overlap. It sounds sort of like video encoding, but it's going to be much less similar and more complicated than adjacent frames in video.

The DNN researchers at Google have done some impressive work with images and image recognition, and I wonder if that could be applied to an approach that uses reference images for specific people, places, logos, etc. and does something like delta encoding for a bunch of images that feature those people, places, logos, etc. At first blush, this seems unlikely to work out – to lead to significant savings – but I think my initial intuition is wrong and that it's just a matter of much better algorithms and constructs.

Another idea I had was what sorts of things we could do at the time of image acquisition – when we're composing and taking photographs – that would help achieve smaller file sizes at a given level of quality. I've never read anyone talk about this before, but there are so many things going on in, say, headshots, product photography, etc. in terms of backgrounds, lighting, colors, color gradients, angles, reflectance, etc. It might be possible to subtly manipulate these things in ways that lead to surprisingly smaller JPEGs, webps, or PIKs without significantly impacting (or even improving) how humans perceive the image.

[Jyrki Alakuijala]

I've also wondered about combining raster and vector approaches in the same format, since it's clear that some images would benefit much more from one or the other.

I also want to see a format that explicitly accounts for image sets in addition to standalone images, since a lot of groups of images on webpages, social media, etc. feature the same settings and subjects and have lots of encodable overlap. It sounds sort of like video encoding, but it's going to be much less similar and more complicated than adjacent frames in video.

The DNN researchers at Google have done some impressive work with images and image recognition, and I wonder if that could be applied to an approach that uses reference images for specific people, places, logos, etc. and does something like delta encoding for a bunch of images that feature those people, places, logos, etc. At first blush, this seems unlikely to work out – to lead to significant savings – but I think my initial intuition is wrong and that it's just a matter of much better algorithms and constructs.

Another idea I had was what sorts of things we could do at the time of image acquisition – when we're composing and taking photographs – that would help achieve smaller file sizes at a given level of quality. I've never read anyone talk about this before, but there are so many things going on in, say, headshots, product photography, etc. in terms of backgrounds, lighting, colors, color gradients, angles, reflectance, etc. It might be possible to subtly manipulate these things in ways that lead to surprisingly smaller JPEGs, webps, or PIKs without significantly impacting (or even improving) how humans perceive the image.

That is a lot of good ideas.

[Jyrki Alakuijala]

Google must have beautiful, rich, and useful data on webp performance in all aspects, but for some reason it refuses to release it.

Post a request to release such data on https://bugs.chromium.org/p/webp/issues/list

My current understanding that WebP lossy is justified by PSNR, Y-SSIM and RGB-SSIM plots.

Guetzli is justified by Butteraugli and our somewhat outrageous claims are made slightly more plausible by a human comparison against libjpeg.

All I've seen are old comparisons between webp and PNGs where they used pngcrush and pngout, for example. But in the wild we're not limited to pngcrush and pngout, and those aren't even the most powerful tools.

In the wild many people use the default settings. Back then when I did that study 4 % of PNGs in the internet were 16-bits per channel. Definitely they were not optimized at all, and needed to be reoptimized to be fairly comparable with WebP lossless.

Back then WebP lossless with default settings gave a 40+ % reduction from PNG file size considering PNGs as they were on the internet (without recompression).

Does webp actually produce smaller files than JPEG and PNG if we use all the tools available for those legacy formats? I don't know.

At high quality JPEG wins over WebP, but PNG wins over WebP for more random reasons. About 2 % of PNGs in the internet cannot be further compressed by WebP lossless. Some part of that 2 % may have gone through a near-lossless PNG conversion (possibly about 0.5 %).

I'm not sure if anyone knows. My hunch is that it probably does, but the difference isn't as big as Google claims in its dated tests on the webp website.

Lossless testing is not difficult. Nowadays you get 26-27 % gain from WebP lossless against ZopfliPNG. ZopfliPNG has improved PNGs (over from the PNGOUT days), but also WebP lossless has improved a few percent in density.

I'm more confident that webp can beat superoptimized PNGs than superoptimized JPEGs, but the JPEG case is the most complicated because of the difficulty of apples-to-apples comparison in terms of "quality".

Yeah. I'm fully with you.

Try compressing a big one megabyte photo down to 15 kB, and you will be convinced that there are some use cases where WebP lossy is far superior to JPEG. If you take graphics images, WebP can be competitive even at higher quality settings.

For high quality photography, I (and butteraugli) believe that JPEG is actually a better than WebP.

We have quality integers that we're throwing around that aren't commensurable across formats and most web development shops don't have enough staff to deep dive into the finer points of image optimization.

This is why I think it is important that default settings don't spoil the image and still compress relatively well.

Last time I checked with butteraugli WebP lossy at quality 85 was somewhat similar to JPEG YUV444 at quality 77, i.e., that was the place where the size and quality curves met. Below JPEG quality 77 WebP lossy wins, above JPEG quality 77 JPEG wins (for photography). At that quality WebP quality 85 would match with JPEG quality 77, so there may indeed be a difference on how to use the quality settings in these two encoders. This was based on the maximum compression artefact in an image -- averaging from 1000 images.

Depending on methodology, which version of butteraugli one uses, or if one uses ssimulacra or ms-ssim on comparison, one can get different equivalence numbers.

WebP's artefacts tend to be more friendly to the eye than JPEG's, so when things start to go south, WebP may be more pleasant.

[SolidComp]

Thanks for rich and thoughtful reply Jyrki. On PNG, do you have any idea how webp fares against Kornel's PNG quant? When it comes to PNG in general, I think people get distracted by the fact that it's "lossless". Yes it's defined and specced as a lossless compression of bitmap data, but I think lossy PNG optimization is often undetectable by users, and is much less problematic than lossy JPEGs. PNGs are simple. (And the hybrid vector/raster image format I have in mind would significantly shrink a lot of PNGs.)

On my idea of reference images and delta encoding, it would be awesome if a few prior photos of a person or mountain made subsequent photos much smaller. This is going to be complicated and hard, but I think in the end it will work.

The DNN team recently did some work on security camera footage enhancement. Security cameras suck, seem to be a decade or more old, with fuzzy pictures. It would be interesting if any specific crime case, they could take reference images of employees (of a store, bank, etc.) and get various footage of those employees from the security cameras. Then analyze the delta between known, high quality photos and grainy security cam footage of those same people. They could then build a model of those specific security cameras' behavior in how they capture and compress a person's face, in great detail, with specific features, morphological patterns, etc. Then they could use this work to help identify criminals by taking a lineup of suspects, getting the kind of quality reference photos previously mentioned, and have their model predict how that specific security camera in the bank or whatever would tend to encode that person's face. If this could be made to be sufficiently reliable, it would be a great tool.

[schnaader]

I've also wondered about combining raster and vector approaches in the same format, since it's clear that some images would benefit much more from one or the other.

Well, there's PDF, SVG and such where you can combine both, though they can't be used to compress a given image automatically.

I also want to see a format that explicitly accounts for image sets in addition to standalone images, since a lot of groups of images on webpages, social media, etc. feature the same settings and subjects and have lots of encodable overlap. It sounds sort of like video encoding, but it's going to be much less similar and more complicated than adjacent frames in video.

WaveOne made some progress to this last year using machine learning, see the paragraph "Domain-adaptive compression" on this page. Additionally, the algorithm compresses "common" objects like faces good at very low bitrates, see the result images in the paper.

[thorfdbg]

JPEG 2000 is worse (10 % more bloaty) than JPEG at high quality for human viewers.

Sorry, that's simply not correct. It is a matter of how you use it and how you drive it. In the language of JPEG, "it is a matter of the quantization matrix". We did all the exercise for the ICIP challenge 2016, and the results should still be available on www.jpeg.org. If you compare a "default JPEG" with its "human oriented quantization matrix" with a PSNR-driven "default JPEG 2000", then you are correct, of course. But using JPEG 2000 this way is just folly. You need to drive the latter in a way that is optimized for human vision, not for PSNR. Equally well, you can tune JPEG towards PSNR of course, with quite some improvements in performance, though it does not become then better visually (actually, worse) and neither better than JPEG 2000.

[thorfdbg]

Thanks for the info. This is interesting. For some reason I thought JPEG-XR was supposed to be better than JPEG 2000 – I think I saw a research paper to that effect.

Yes, I know the paper, and that was simply due to MS not understanding what they were doing: They measured PSNR of a PSNR-optimized JPEG XR codec against PSNR of a visually optimized JPEG 2000 codec (Kakadu). So yes, results are of course "as expected" (or "as wished"), but that is certainly not a scientifically valid comparison. Again, see the ICIP challenge.

JPEG-XR will probably die off soon since Microsoft has been so lazy with it, and no one else is exploiting its headroom, HDR and other features, etc. or building better encoders, e.g. a libjpegxr-turbo or MozJPEG-XR or a Guetzli approach.

Well, not really. MS is using it (at least according to Rico Malvar, head of MS research) internally in their RDP products, and there is also recently an activity going on on the MPEG side to allow a royalty free codec in the HEIF container. HEVC I-frame has IP problems, so they are including JPEG 2000 and JPEG XR there.

If JPEG 2000 is better, why don't people use it? It's always seemed like a confusing and difficult format to adopt. Just viewing JPEG 2000 images on Windows 10 is difficult. I had to do some hacking to help my girlfriend view her MRIs in some format called DICOM where some of the images were JPEG 2000. Is it patent encumbered? I remember hearing something about that.

People *do* use it. Just not in digital photography, where it is more advantageous for the vendors to push their own proprietary codecs to "lock in" their customers to their products. This is "raw" (which is not a format). JPEG 2000 is used a lot in digital cinema and the professional market.

Ancillary question: Do you happen to know if Safari on macOS and iOS actually displays JPEG 2000 files? It's supposed to, but I've never been able to find any mention of it in Apple's documentation, and I have no devices to test. That also makes me wonder if patent issues have prompted Apple to keep quiet about it.

I would believe Apple products are more likely to support JPEG 2000 these days, more than any other vendor.

[Jyrki Alakuijala]

We did all the exercise for the ICIP challenge 2016, and the results should still be available on www.jpeg.org.

Yes, we are both correct. All depends on what constitutes 'high quality'. For me, high quality is something like JPEG YUV444 at quality 95-99. In this range it is possible to find settings that allow for no visual degradation.

In rough terms of bpp, I can get high quality at 3.5 bpp with libjpeg JPEG YUV444, 2.5 bpp with Guetzli, and (currently) with 1.58 bpp with PIK.

ICIP 2016 challenge showed subjective results only up to 1 bpp. For JPEG, 1 BPP corresponds to quality range ~80 and YUV420. I don't consider this high quality, quality 85 YUV444 is already what I consider 'middle' image quality where compression artefacts are universally present.

[Jyrki Alakuijala]

Thanks for rich and thoughtful reply Jyrki. On PNG, do you have any idea how webp fares against Kornel's PNG quant?

Kornel's PNG quant is the best-in-class tool for palettization -- great balance of features and superior quality. Unfortunately there is nothing like it for WebP, and possibly the best you can do is to apply it for a PNG and then convert that PNG to a WebP.

It is possible that one could use Zoltan's greedy palettization and top that with butteraugli to select between the palettized and non-palettized variations for even some more savings in WebP lossless -- but no such tool exists for now.