Universal data detector?

[lunaris]

Is it possible to create an universal data detector?


For example,the data detector scan all the entire file searching for compressible/uncompressible patterns like file type detection (zip and jpeg files which are uncompressible and wav type files inside of file) or data patterns (big repetitions like srep does ,except it will not compress the file).


And after the scanning ,the data detector generates a instruction file (or preprocessed file) to how the compressor must apply the compression in the most efficient way.

[Matt Mahoney]

No (proved by Kolmogorov).

[nburns]

No (proved by Kolmogorov).

You can't create an optimal universal data detector, as Matt said. If you can tolerate a heuristic-based one, then of course it's possible. There are archive utilities and compressors that attempt to recognize compressible versus non-compressible components and act accordingly.

[Kennon Conrad]

zip and jpeg files which are uncompressible

Many .zip files are compressible. I think the simplest method is unzip->zpaq.

[lunaris]

You can't create an optimal universal data detector, as Matt said. If you can tolerate a heuristic-based one, then of course it's possible. There are archive utilities and compressors that attempt to recognize compressible versus non-compressible components and act accordingly.

Precomp for example ,it detects lot of zipped file types and some deflate streams.Srep detects huge distance repetitions .So I think precomp can detect another type of files (like wav and apply flac or wavpack on them or do nothing,just detect).Precomp can be extended for a lot of files.

I think data detection is a form of opposite of compression.Data detection tries to make the file more compressible(file more simple to understand).And compression makes the data less compressible(files more hard to understand)

But it depends of compressor used and (type) of intelligence of user on analisyng the data.Some people thinks multiplications are more simple than additions.

[lunaris]

"Many .zip files are compressible. I think the simplest method is unzip->zpaq. "



That's what I was trying to explain.They are compressible(or better compressible) only on uncompressed forms.

[SoraK05]

Is it possible to create an universal data detector?


For example,the data detector scan all the entire file searching for compressible/uncompressible patterns like file type detection (zip and jpeg files which are uncompressible and wav type files inside of file) or data patterns (big repetitions like srep does ,except it will not compress the file).


And after the scanning ,the data detector generates a instruction file (or preprocessed file) to how the compressor must apply the compression in the most efficient way.

http://encode.su/threads/2006-Hello-...e-an-algorithm
This algorithm has information to preprocess and scan a file for specific areas to specifically work on in advance than on the whole file in one go to be more specific to file distribution as well as compressible/uncompressible areas.

A bit more info in this:
http://encode.su/threads/1987-Random...ll=1#post39529