I have an idea for a multi-threaded compressor

[Earl Colby Pottinger]

My question, if I publish the "C" code here, how much faster does it have to be than other compressors in it's class to be worthwhile looking at?

The compressor is a simple byte oriented compressor/decompressor for a RAM DISK that is why I was looking at speed rather than compression ratio, but I also know multiple threads tend to clog up the caches and main memory interface so that is a major diminish in returns as you increase the number of threads.

Or am I just wasting my time talking about it here?

PS. If the idea of tacking meta data at the beginning of a compressed file already has been done could someone give me some pointers/URLs.

[SolidComp]

You should publish it (on GitLab or Bitbucket). We can't know anything about it until you publish it and benchmark it (or others benchmark it). It only has to be slightly faster to be worth looking at. Hell, it might be worth looking at even if it's slower than its peers, if it has other strengths like CPU/power efficiency, or compression ratio, or whatever. We can always learn something from any codec anyway. Multithreaded compression is still not mature or even common in some quarters, so a new multithreaded implementation is always nice. Anyway, fortune favors the brave – don't be afraid to put yourself out there.

What do you mean by metadata at the beginning of the compressed file? That sounds like it could be a header, a model, a dictionary, or something else. What kind of metadata?

[necros]

IMHO if you want it to be general purpose oriented it should compress slightly faster and tighter than 7z 1st of all.

[JamesB]

If you're focusing on things like RAM disk and byte oriented packing, then likely comparing to LZ4 and/or Snappy are the obvious existing contenders out there.

Give those a whirl on some data and see how your one compares.
Good luck.