Neural Image Compression

[Shelwien]

I'd like to share a side project I worked on which generalizes transform coding to the nonlinear case. Here the transforms are represented by neural networks, which learn the appropriate form of the transform. The result of the transform is then quantized using standard entropy coding.

Github: https://github.com/Justin-Tan/high-f...ve-compression
Interactive Demo: https://colab.research.google.com/gi...lab_demo.ipynb

There are some obvious shortcomings to this method - such as, that it only caters for image data, cannot be adjusted to attain a variable bitrate, short of training a different model, and is unrealistically slow for practical applications.

I'm not a traditional compression expert, so would appreciate any insight about the deficiencies of this method from those who are. Note this is not my original idea and is a reimplementation.

[pklat]

it seems nvidia is already doing it with live video:
https://arstechnica.com/gadgets/2020...dwidth-by-10x/

[Jyrki Alakuijala]

There is a conference on the topic at http://www.compression.cc/

It has been organized since 2018 and participants to that conference represent the current-state-of-the-art.

It is relatively easy to participate in their automated compression competition that is organize about 2 months before the conference.

The best effort from the compression density/visual quality viewpoint that I know of is https://hific.github.io/

Most such efforts are not practical with current hardware, and the next challenge in the field is to get some of the benefits into a practical form. We have attempted to get some of those benefits into JPEG XL. Specifically, I plan to use a gradient search at encoding time, but more traditional -- but larger, selectively layered and overcomplete -- transforms at decoding time.

[pklat]

so the problem now is that compression artifact are 'unpredictable'? you can't tell how much or in what way the image has been changed/distorted unless you have original?

[Jyrki Alakuijala]

so the problem now is that compression artifact are 'unpredictable'? you can't tell how much or in what way the image has been changed/distorted unless you have original?

In some use cases it is a problem, in others it is a blessing.

Creative compression can be bad in verification and validation imaging, medical imaging, or recording a crime scene.

In content creation such as game textures it is just good that the system fills in the gaps. In a quick selfie it can be ok that skin imperfections are erased.

It all comes down to system utility and fairness. If the system utility and fairness are increased overall, then a technology may be good to deploy. Certainly the neural compression systems can increase the compression density.

[pklat]

neural networks could probably be very good at 'restoring', say, screenshots from scanned old computer magazines.
They could recognize windows, icons, mouse pointer, fonts, etc... reconstruct it while compressing.

[zyzzle]

neural networks could probably be very good at 'restoring', say, screenshots from scanned old computer magazines.
They could recognize windows, icons, mouse pointer, fonts, etc... reconstruct it while compressing.

Well, that's a tall order with current hardware, but perhaps neural nets on quantum computers could be made to excel at this type of task, ie cleaning up scanned documents, magazine articles, and low-resolution material. Looking forward to that day, but it might still be a full generation off (ie, 20 years away). But, by then compression of images won't matter as much, as we'll be into yottabyes and Zetabytes on your fingertips.

[pklat]

OCR is probably already better than human, captchas for feeding AI haven't used it for a long time now.
There are a lot of combinations (fonts, styles, DTP software etc...) but not infinite, so I think it could reconstruct original .qxp or .doc or whatever with good enough accuracy.
AI can also be used for removing hard-coded subtitles and similar stuff from video, images.
And removing (ingress) noise, scratches from audio/video that can't be compressed.

[Nania Francesco]

I wanted to make a practical comparison with other formats that I do not cite for fairness. I think it's really interesting to make it at the cost of bytes. My question is whether there is an encoder and decoder del formato High-Fidelity Generative Image Compression ?

[Nania Francesco]

I tried to install with python all the various gadgets and various libraries (pip.. etc) following the guide provided by the programmer. Honestly he finds many errors even during the installation of packages (torch.. etc.). I don't understand why they used python?!

[pklat]

they all use/need tensor flow, and that uses python.

I think you usually need some high end card with a lot of CUDA cores and even with that training the network takes too long to use your own data.