Compression techniques that don't require repetition

[Bundle]

So, the compression algorithm I've been working on doesn't require ANY repetition to be effective. Currently, it's crunching runs of 1,024 bytes down to a 20-byte representation, and that's really squeezing everything out of it. I could stand to raise the representation up to even 512 bytes and still be getting great performance out of it. On a 1GB "random data" file, it compressed at 549 Mbps in 18.7 seconds at 98.24% compression. The compressed file is ~18.5kb now. The decompressor is already coded, but I am working out bugs. I lose a few bytes on decompression, so it's close, but not 100%.

I will say in advance, I understand any doubt. I think it's healthy to be skeptical, especially considering this would break some rules of compression.

All this to ask, are there any current compression algorithms that don't rely on repetition or patterns?

[Shelwien]

> so it's close, but not 100%.

Its not a problem, you can always just use a binary diff tool: https://github.com/sisong/HDiffPatch/releases
(diff correct file with partially decoded file, then count diff size as part of compressed size).
or alternatively ECC: https://en.wikipedia.org/wiki/Parchive

> are there any current compression algorithms that don't rely on repetition or patterns?

Depends on what you mean by "repetitions".
Entropy coding algorithms don't rely on repeated long strings, but do rely on statistics, ie patterns in symbol repetitions.

> break some rules of compression.

Its not about "rules" or anything.
Its more about ability to count the numbers of possible input files and compressed files.

[Bundle]

Thanks! I'll check out the binary diff tool. It will save me loads of time trying to find the error.

As far as repetition, I meant patterns or frequencies. My algorithm uses arithmetic but no probabilities or anything.

Did you mean count how many files can be compressed to the same compressed sequence?

[Shelwien]

> Did you mean count how many files can be compressed to the same compressed sequence?

Its obvious that its impossible to decompress different data from the same "compressed sequence".

> As far as repetition, I meant patterns or frequencies.

Its possible to replace the entropy coding with hashing, but in the end, "patterns or frequencies" are the only sources of compressible redundancy.
https://en.wikipedia.org/wiki/Kolmogorov_complexity

[Bundle]

Ah, I see. It says there is a lower limit for the size of the compressor able to compress a file, if I'm reading that right. The compressor and decompressor are in the same file and it's about 15kb.

[Shelwien]

Yes, its relevant for contests like this: http://prize.hutter1.net/

[Bundle]

Yes, its relevant for contests like this: http://prize.hutter1.net/

And I wouldn't mind doing the challenge, but I don't want to give away the source code.

[xinix]

And I wouldn't mind doing the challenge, but I don't want to give away the source code.

You are not required to source code compressor
Only decompressor.
Or:
Well then https://marknelson.us/posts/2012/10/...turns-ten.html
That I need to give the source code.

[Bundle]

You are not required to source code compressor
Only decompressor.
Or:
Well then https://marknelson.us/posts/2012/10/...turns-ten.html
That I need to give the source code.

But giving the decompressor source code pretty much gives away the method. I haven't decided what I want to do with the algorithm yet.

[Bundle]

Yes, its relevant for contests like this: http://prize.hutter1.net/

I looked at the Hutter Challenge, and I was able to compress the 95.3MB enwik8 file down to 2.03MB.

[Sportman]

2.03MB.

2,128,609 bytes?

Can be $43,854 price when your output file and your decompressor file lead to identical enwik8 (and a lot of jobless people in this forum).

[hexagone]

I looked at the Hutter Challenge, and I was able to compress the 95.3MB enwik8 file down to 2.03MB.

Nope.

[Sportman]

You can create a version only working at 1MB files, with only 3 bytes or more compression gain and submit it in this thread to claim my reward:
https://encode.su/threads/1176-losel...ll=1#post59199

[Bundle]

Nope.

That's not a very detailed response.

[hexagone]

That's not a very detailed response.

It was not a response, since you made a statement. but, ok, prove me wrong.

[Shelwien]

Compressing any files to any size is not a problem.
Correctly decompressing them is, though.
It looks like another case of misunderstanding how hashing works, based on first post.

Also, as to Hutter Prize - I'm pretty sure that they would accept a 2M entry even without sources.
Its a new rule anyway, it didn't exist before, and nobody submitted sources yet.