Optimizing image compression codecs for object recognition / a 21st century format

[SolidComp]

Hi all,

Here's an interesting article

about moving object recognition neural networks directly into a camera's optics.

This reminded me that our next-generation image format should be optimized for object recognition, machine vision, etc. Among other things, that is. We still need better compression, fast encode and decode, etc.

I think that if you're working on a new image format in 2018, you should think about how machine vision and object recognition work, and the various ways an image format could make those workloads easier.

I also think that it's time for a new acquisition format. It's time to replace JPEG. Recent formats like webp have been focused on the web, but it would be better if a new image format was designed as an acquisition format – the format the camera would encode when a photo is taken. It will be easier to replace JPEG and get wide adoption if a new image format were carefully and rigorously designed to replace JPEG at every stage, from acquisition, professional work, the web, etc.

And a new format should be GPU-only or GPU-accelerated. The platforms we're going to care about will all have GPUs. We don't need to worry about really old devices – they can continue to use JPEG or webp. GPU formats will be much more efficient than CPU-driven formats. It's such a waste of energy to use CPUs for imaging. (The cameras themselves use ASICs, basically, to encode. And if there's any way to design an image format such that it's easier or cheaper to build ASICs for, we should do that.)

If all these goals are compatible, great. If not, oh well. But I think Google, Apple, Microsoft, and others (Dropbox?) should form a team that is committed to excellence, a team that is driven to develop a great new image format for the 21st century, a team that will start with a clean sheet and kick ass. We need this because we're wasting too much bandwidth and storage with JPEG. The small, half-hearted efforts we've seen so far are unlikely to result in a format that replaces JPEG, and therefore will leave us with lots of wasted storage space and bandwidth.

[Jyrki Alakuijala]

If not, oh well. But I think Google, Apple, Microsoft, and others ...

This team exists. Their work is pretty good and ready. It is called av1. It is complex enough that in practice it can be run only with special hardware, but not with gpus as far as I know. Special hardware can be more energy efficient than either gpu or cpu.

The deep learning research that you can read about usually comes with a small model of 777 MB to 3.33 GB that is necessary for decoding, and covers only one domain area such as cityscapes from Stockholm in spring 2017.

[SolidComp]

This team exists. Their work is pretty good and ready. It is called av1. It is complex enough that in practice it can be run only with special hardware, but not with gpus as far as I know. Special hardware can be more energy efficient than either gpu or cpu.

The deep learning research that you can read about usually comes with a small model of 777 MB to 3.33 GB that is necessary for decoding, and covers only one domain area such as cityscapes from Stockholm in spring 2017.

Didn't you say something about how still images from AV1 are not that good? Or was that HEVC?

What about PIK? Is it based on AV1?

[Jyrki Alakuijala]

Didn't you say something about how still images from AV1 are not that good? Or was that HEVC?

What about PIK? Is it based on AV1?

AV1 and HEVC are pretty good on low BPP (around ~0.5 BPP).

PIK seems to work better than AV1 and HEVC at high BPP (~1.0 BPP and above).

PIK is much faster to decode on software.