Exogenous decompression

[SolidComp]

Hi all – I've been thinking about exogenous decompression lately. I created this term to describe a scenario where a decompressor can better restore lossily compressed data by using exogenous information, such as characteristics of the encoder/compressor.

For example, with a video stream maybe the decompressor knows which encoder was used and can use this information to achieve a better result than standard decompression. Maybe there are specific characteristics and quirks of the encoder that would be fruitful to know at decomp time.

Does this sound feasible? This is slightly related to my idea to use exogenous metadata to better compress audio/music, but this is purely about decomp.

I don't know enough about video codecs to know specifically what sort of information might be helpful.

Relatedly, a Google project gave me an idea for video streams. When there's footage from, say, a security camera, perhaps it could be enhanced by knowing specifics about the camera model and its encoder. Even more interesting would be to:

1. Take known objects, like employees, get them on the camera
2. Relate that camera footage to high res imagery (video or photo) of those same employee faces
3. Figure out how the camera encodes those known objects
4. Use that to do the reverse – enhance camera footage of unknown targets (e.g. criminals) by understanding how the camera encodes known examples

Thoughts?

[JamesB]

There are various video players with options that attempt to correct for mpeg artifacts, eg ringing and blocking. That's essentially doing the same trick, but it's more of a format correction than an encoder specific correction.

Edit: allegedly the Huawei P30 Pro moon pics are corrected for using predefined moons pics! That's a rather exceptional case of a rendering algorithm correcting for loss if so, but the company disputes it.

https://www.androidauthority.com/huawei-p30-pro-moon-mode-controversy-978486/

I have to laugh though: "To test out this theory, Yue photographed the moon from several different angles as well as photographed non-moon objects using Moon Mode." Different angles! Is he an astronaught? I guess he could have meant different phases and maybe it's just a bad translation.

[Gonzalo]

This seems to be akin to the AI field. If I understand you correctly, you're not talking about some kind of off-line dictionary like BARF and its built-in Calgary Corpus but actual reconstruction AKA imagination. This off course can only be applied to lossy formats. There are several attempts to do so right now. Just google 'deep learning image reconstruction' or similar and you'll find quite a few.
I suppose a properly trained network could learn exactly what artifacts are introduced by a particular camera model or codec and find a way to compensate for it. Here is one example of image reconstruction using neural networks, extracted from the article 30 AMAZING APPLICATIONS OF DEEP LEARNING:


"

3. Pixel restoration CSI style

In the show CSI they often zoom into videos beyond the resolution of the actual video. This seemed completely unreliable and there are even a few videos on YouTube like the one below where people explain they don't watch CSI because that is unrealistic.



Well, it was unrealistic until Deep Learning. Early in 2017, Google Brain researchers trained a Deep Learning network to take very low resolution images of faces and predict what each face most likely looks like. They call the method Pixel Recursive Super Resolution which enhances resolution of photos significantly. In the image below you can see the original 8x8 photos, the ground truth (which was the real face originally in the photos) and in the middle the guess of the computer. Obviously it is not perfect, as it cannot be, but it is pretty unbelievable that the computer can guesstimate so well many of the features of the person in the photo.
8x8 pixel photos were inputted into a Deep Learning network which tried to guess what the original face looked like. As you can see it was fairly close (the correct answer is under "ground truth").


"

[Jarek]

There are lots of options here, e.g.:
1) rendering - decoders have models of among others actors and render entire movie - ultimate video compression,
2) ML based upscaling, especially using GANs: upscaler trains to improve image such that discriminator cannot distinguish it from training dataset,
3) lots of information theory-based possibilities, especially Wyner-Ziv ( https://en.wikipedia.org/wiki/Distributed_source_coding ) - encoder can build on information known only to decoder, for example:
- https://en.wikipedia.org/wiki/Free_viewpoint_television - only decoder known in which direction the person is actually looking,
- universal updates - encoder doesn't need to know versions of individual decoders,
- entropy coding where only decoder knows statistical model.

ps. Gonzalo, it is probably GAN-based trained on CelebA detaset - will generate nicely looking celebrity faces, but usually far from real face of common person. Like searching dataset for closest face.

[SolidComp]

There are various video players with options that attempt to correct for mpeg artifacts, eg ringing and blocking. That's essentially doing the same trick, but it's more of a format correction than an encoder specific correction.

Edit: allegedly the Huawei P30 Pro moon pics are corrected for using predefined moons pics! That's a rather exceptional case of a rendering algorithm correcting for loss if so, but the company disputes it.

https://www.androidauthority.com/huawei-p30-pro-moon-mode-controversy-978486/

I have to laugh though: "To test out this theory, Yue photographed the moon from several different angles as well as photographed non-moon objects using Moon Mode." Different angles! Is he an astronaught? I guess he could have meant different phases and maybe it's just a bad translation.

Interesting, I had heard about the P30 having some unique features but not that.