Superduper - Precompressor

[Lucas]

Hi guys, here's my first actual compressor, my goal was to make a faster and more memory efficient utility like srep, superduper is just short for "super deduplication" (it sounded snazzy). Its performance is pretty good, with encoding generally around 200 MB/s and decoding near 2 GB/s on an overclocked pentium g3258.

https://github.com/loxxous/Superduper

Let me know if you encounter any bugs.

[SolidComp]

I admire anyone who writes their own compressor. Could you elaborate on what it does or what workloads/content it's geared toward? I've struggled to understand what precompressors actually do, and how they fit into a typical compression workflow. I assume there's no such thing as a universal precompressor, so Superduper is pre.. what? What happens next after using Superduper to precompress something – what sort of compression would follow?

[Lucas]

I admire anyone who writes their own compressor. Could you elaborate on what it does or what workloads/content it's geared toward? I've struggled to understand what precompressors actually do, and how they fit into a typical compression workflow. I assume there's no such thing as a universal precompressor, so Superduper is pre.. what? What happens next after using Superduper to precompress something – what sort of compression would follow?

These kinds of preprocessors are generally geared towards statistical context models, the point of SD is to remove long range redundancy fast as possible, my intent with this is to have it used as a precompression scheme to a contex model. the idea is simple: less data to model, the faster the compression. on enwik9 it can reduce the 1GB file by 17%, that's 17% less data a context model needs to read to achieve pretty much the same compression ratio. It's about finding ways on saving time on the most complex routine.
Also worth noting that lz77 is heavily asymmetric, decoding has very little overhead and will almost always speed up decompression routines.

[xinix]

Compile by Krinkels.org

[Zonder]

Atm it looks like some Jokeware. v0.02 tested:

duped1.tar 5757Mb:
1440Mb srep+lzma2:d16mb
2054Mb lzma2:d16mb
2149Mb superduper+lzma2:d16mb
3666Mb srep
4511Mb superduper

duped2.tar (2 same uncompressible files tared) 4686Mb:
2341Mb srep
4657Mb superduper

[Lucas]

Atm it looks like some Jokeware. v0.02 tested:

duped1.tar 5757Mb:
1440Mb srep+lzma2:d16mb
2054Mb lzma2:d16mb
2149Mb superduper+lzma2:d16mb
3666Mb srep
4511Mb superduper

duped2.tar (2 same uncompressible files tared) 4686Mb:
2341Mb srep
4657Mb superduper

I'm surprised it even compressed that second test file of yours.
Right now it uses a 2 MB window and a small hash (hence the low memory usage). I'm still working on it. The most it can theoretically support are 256 MB distances, I'll try to get that working tonight.

Update: new release up on github, now with user selectable block size and memory for the hash.

[SvenBent]

excuse my naive understanding. but if it only compress "identical parts" over a 256mb window. is it really a deduplikation ?
I thoguht the deduplication was meant for when it works across huge data size. this is even less than 7-zip

[Lucas]

Deduplication is kinda broad, it's just removal of identical blocks, which is just lz77, that was the original name to this project but it wasn't unique so I changed it. This is actually a sub model to a project I'm working on, the main compression model I use can't see identical strings because it can only make predictions rather than directly encode strings, so on some files it benefits to use lz77 preprocessing. I release my work in parts for people to play around with because that's how I learned how different algorithms perform in different orders/stages. Like with raw bwt transformations follwed by fiddling around with second stage algorithms to try and find ways to improve performance or compression ratio.