how the brain compresses images

[willvarfar]

a popular-science intro to some newly published research: http://www.theregister.co.uk/2011/02/11/brain_jpegs/
what would be interesting is lossy compression that favoured this, or as input into object recognition problems. But perhaps this is all so obvious everyone is already on top of it.

[EwenG]

The brain uses a "modelling" method of remembering things. We all know what a woman likes without clothing, but we "forget" about the little details such as moles, variants in skin tones & such - unless it's one of them hairy moles, which the brain tends to record. We know the shape (model), we just don't recall ALL the details in precision. Likewise, we remember what the ball we throw at the dog looks like, but we tend to ignore a lot of the detail (exact teeth marks, imperfections in the colouring and such). However, we don't eliminate ALL the imperfections.

I agree that an encoding method would be nice that could break down the "basic" shapes found in an image, eg: squares, circles, triangles etc. and record a certain amount of the "imperfections" of the image.

I deal a lot with 3D renderings. The biggest challenge in that is creating imperfections in the image. A good example being the skin, which is NEVER perfect. So an image of someone with great skin looks like what we want people to see, but most of the time it's "hmm, there's something about this image that's not quite right" - we get that "niggling" feeling something is missing. So we spend a LOT of time creating tonnes of files with "imperfect" skin, and use that texture on the final passes. It takes me hours on perfecting the imperfections, and i've only had a few "good" results so far. I ramble again.

An engine that not only breaks down the shapes, but can recall certain degrees of imperfection (the lossy part) would be a GREAT boon to the art world in general. To my knowledge there's no such beast as yet. No doubt many have thought of it though.

Hope that made some kind of sense.

[Matt Mahoney]

Very interesting. Landauer's studies of long term memory ( http://csjarchive.cogsci.rpi.edu/198...p0493/MAIN.PDF ) suggest we remember at a rate of around 2 bits per second. So in theory, you could compress video to this rate using a model of your brain (which of course we don't have). One way to do this would be to encode a movie as a script and compress it. The decompresser would have to be able to read the script and construct a movie close enough to the original that you wouldn't notice the difference. It's an AI problem.

We do have models of the simpler or lower layers of perception, which lossy compressors already exploit. For example, we can quantize audio samples on a logarithmic scale to match the response of our ears.

[EwenG]

Actually - i'll continue my rambling for a bit as I think this is "kind of" relevant.

I use Poser for 3D rendering. The file sizes it uses is hideously over bloated for various reasons (mostly badly encoded JPEG/PNGs). A single model can contain up to 3 x 4000x4000 JPEG files which are the main texture itself, a bump map (greyscale map showing where the detail is raised in the texture), and a transparency map (greyscale showing where opacity is used). Sometimes there's a template file too. All these JPEGs are USUALLY similar in shape - so an engine could recognise the similar models in the files, and create a better compressed file - very much similar to encoding a film.

Another sore point in these images is "eyes" for example. Usually the only difference is the colouring of the iris - however each one creates a file that is over 1Mb for such a tiny bit of change.

Maybe I should be thinking along the lines of "motion-picture" type encoding on these though ?

[quick edit]
Seems I posted this just before I saw Matts post above. For a single file, not sure if it's worthwhile, but in a "collection" of images it might be VERY useful indeed. Certainly worth thinking about.

[Piotr Tarsa]

Matt:
I think the problem is that human focuses on some very small set of details. And every time on a different set. So using brain compression we can store something like memories, not actual sensations.

[Matt Mahoney]

True. To get 2 bps video the compressor would need to know what part of the video you are likely to pay attention to and remember. Each person would need a custom compressor based on a model of their brain. Not likely anytime soon.

Still, I think there is much room for improvement using AI techniques. Speech could be compressed to 10 bps by converting it to text if we had better speech recognition and speech synthesis. Both are hard problems.

[willvarfar]

random thought:

1. record audio
2. speech to text conversion and timing data
3. then text to speech conversion
4. then store the text, the timing data and a lossy difference between the text-to-speech conversion and the recorded audio.

Is this approach been tried? Any bad results to dissect?

[Shelwien]

@willvarfar: So what you're expecting to get like that? If its perfect, maybe the listener won't notice that speech is synthesized,
and would acquire the same information as with original record.
Current implementations are far from that, though.
But its also kinda obvious that its not a matter of "lossy difference" - you won't be able to align the text-to-speech signal
with original so easily, and the difference would be likely less compressible than original.

Well, it may be possible to use some kind of phonetic transcription as a context for compression of speech signal,
which is somewhat similar to what you suggest, but imho syntesized speech alone would be much less useful
as a context than transcription.

And as to differences, I think you can test it with a simpler framework first - get a good modern speech codec
to encode/decode your recording, then subtract it from original and try compressing it.
(there're console codecs like that in Intel IPP if you don't know what to use).

[lunaris]

True. To get 2 bps video the compressor would need to know what part of the video you are likely to pay attention to and remember. Each person would need a custom compressor based on a model of their brain. Not likely anytime soon.

Still, I think there is much room for improvement using AI techniques. Speech could be compressed to 10 bps by converting it to text if we had better speech recognition and speech synthesis. Both are hard problems.

I think this is not a good way to follow. I believe the language is a simple subset what we call "image" . "A" is just a image.

The good way is use more image communication than trying to convert everything for language.

OFF:Extraterrestrials use image-like languages!

Someday I will se a very easy computer language which uses only images to program.

[Shelwien]

> Someday I will se a very easy computer language which uses only images to program.

http://en.wikipedia.org/wiki/Piet_(programming_language)

The whole idea with images for communication is really bothersome though, imho.
Like when people make GUIs with colorful icons, but there's no way to find out what they do without clicking them all.

[lunaris]

> Someday I will se a very easy computer language which uses only images to program.

http://en.wikipedia.org/wiki/Piet_(programming_language)

The whole idea with images for communication is really bothersome though, imho.
Like when people make GUIs with colorful icons, but there's no way to find out what they do without clicking them all.

Yes,but images is a LOT more expressive and produces a easy way to program things very quickly .

Piet is very simplistic .It's a 2d language. But it's a good progress because it includes color . And does not use lot of geometry like cubes, spheres and all 3d forms.

Textual Programming languages should include color in syntax.

[Shelwien]

> Yes,but images is a LOT more expressive and produces a easy way to program things very quickly .

1. google defines expressive as "Effectively conveying thought or feeling".
And in compression context "effective" is supposed to mean shorter description length.
Now, 16x16 RGB icon is 768 bytes.
Do you really think that 768 bytes of english plaintext is less expressive than a 16x16 icon?

2. There're too many compatibility issues. People use XML instead of binary container formats
(eg. RIFF is flexible enough to replace XML, and would be much more compact) because there're
issues with supported charsets in viewers, editors, and even communication protocols.
And there're even worse "compatibility issues" - http://en.wikipedia.org/wiki/Color_b...s#Epidemiology

[lunaris]

> Yes,but images is a LOT more expressive and produces a easy way to program things very quickly .

1. google defines expressive as "Effectively conveying thought or feeling".
And in compression context "effective" is supposed to mean shorter description length.
Now, 16x16 RGB icon is 768 bytes.
Do you really think that 768 bytes of english plaintext is less expressive than a 16x16 icon?

2. There're too many compatibility issues. People use XML instead of binary container formats
(eg. RIFF is flexible enough to replace XML, and would be much more compact) because there're
issues with supported charsets in viewers, editors, and even communication protocols.
And there're even worse "compatibility issues" - http://en.wikipedia.org/wiki/Color_b...s#Epidemiology

There are problems of course,but using colors and forms to program computers helps designers create programs easily.

SVG for example is a lot more easy to draw a circle using a illustrator than writing in a xml file.