Image compression and end-user display resolution

[SolidComp]

Hi all -- Typical smartphone-acquired images are 8 - 16 megapixels, with some up to 20. Typical display resolutions on smartphones and desktop/laptop are 2 - 4 megapixels, with 4K reaching 8 megapixels, and 5k approaching 15 megapixels (only 1 megapixel for most iPhones). (Both 4K and 5K displays are still quite rare. Fewer than 10 percent of desktop users have them.)

I don't understand how downscaling actually works on displays. For example, I don't know how a 1080p video stream is scaled to a 720p display, or how a 16 megapixel photo is scaled to a 3.7 megapixel display (1440 × 2560, e.g. many Android flagships).

Questions:

1) Do different image compression codecs yield different behavior or tendencies when downscaled?

2) I think all or most of our image comparison tests are on downscaled images. Is this a problem?

3) Would it be helpful for compression codecs to be aware, in some sense, of downscaling / end-user resolutions when they compress images? Could images be optimized for downscaling, or optimized for the bounds of end-user display resolution?

[Jyrki Alakuijala]

Hi all -- Typical smartphone-acquired images are 8 - 16 megapixels, with some up to 20. Typical display resolutions on smartphones and desktop/laptop are 2 - 4 megapixels, with 4K reaching 8 megapixels, and 5k approaching 15 megapixels (only 1 megapixel for most iPhones). (Both 4K and 5K displays are still quite rare. Fewer than 10 percent of desktop users have them.)

I don't understand how downscaling actually works on displays. For example, I don't know how a 1080p video stream is scaled to a 720p display, or how a 16 megapixel photo is scaled to a 3.7 megapixel display (1440 × 2560, e.g. many Android flagships).

Questions:

1) Do different image compression codecs yield different behavior or tendencies when downscaled?

2) I think all or most of our image comparison tests are on downscaled images. Is this a problem?

3) Would it be helpful for compression codecs to be aware, in some sense, of downscaling / end-user resolutions when they compress images? Could images be optimized for downscaling, or optimized for the bounds of end-user display resolution?

Original resolution images have correlations that originate from the algorithm used for demosaicking. Resampled images don't have these artefacts. Commercial websites like Amazon use resampled images. Image sites like Flickr, Instagram and Facebook resampled images, too.

[thorfdbg]

1) Do different image compression codecs yield different behavior or tendencies when downscaled?

Very likely so. If you downscale, correlation lengths shrink as well, and the amount of correlations goes down as well.

2) I think all or most of our image comparison tests are on downscaled images.

Huh? Not here, at least. Tests are made on images "as close as the original" because that is a primary use case.

Is this a problem?

That depends on what the codec should be good for. If it is for delivery of content over the internet, less so. If storage of your photo collection on an SD card, likely so.

3) Would it be helpful for compression codecs to be aware, in some sense, of downscaling / end-user resolutions when they compress images? Could images be optimized for downscaling, or optimized for the bounds of end-user display resolution?

Probably. Some codecs offer downscaling as part of their design (JPEG 2000 does). Taking advantage of correlations due to demosaiking is something one can think of, but I believe that this happens more or less already "by itself" by any decent algorithm.

[Jyrki Alakuijala]

Taking advantage of correlations due to demosaiking is something one can think of, but I believe that this happens more or less already "by itself" by any decent algorithm.

For lossy, yes. For lossless algorithm you choose which decorrelations to model because you pay the price in d coding speed. BCIF models the demosaicking results best in my experience. I didn't consider it as important and I'm willing to take a 5–10 % density hit due to bad modeling of demosaicking artefacts in exchange of 10x faster decoding. Sending camera resolution lossless images is rarely a low latency activity anyway.