Results 1 to 4 of 4

Thread: I have an idea for a multi-threaded compressor

  1. #1
    Join Date
    Mar 2010
    Thanked 2 Times in 2 Posts

    I have an idea for a multi-threaded compressor

    My question, if I publish the "C" code here, how much faster does it have to be than other compressors in it's class to be worthwhile looking at?

    The compressor is a simple byte oriented compressor/decompressor for a RAM DISK that is why I was looking at speed rather than compression ratio, but I also know multiple threads tend to clog up the caches and main memory interface so that is a major diminish in returns as you increase the number of threads.

    Or am I just wasting my time talking about it here?

    PS. If the idea of tacking meta data at the beginning of a compressed file already has been done could someone give me some pointers/URLs.

  2. Thanks:

    snowcat (1st January 2019)

  3. #2
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Thanked 54 Times in 38 Posts
    You should publish it (on GitLab or Bitbucket). We can't know anything about it until you publish it and benchmark it (or others benchmark it). It only has to be slightly faster to be worth looking at. Hell, it might be worth looking at even if it's slower than its peers, if it has other strengths like CPU/power efficiency, or compression ratio, or whatever. We can always learn something from any codec anyway. Multithreaded compression is still not mature or even common in some quarters, so a new multithreaded implementation is always nice. Anyway, fortune favors the brave – don't be afraid to put yourself out there.

    What do you mean by metadata at the beginning of the compressed file? That sounds like it could be a header, a model, a dictionary, or something else. What kind of metadata?

  4. Thanks:

    snowcat (1st January 2019)

  5. #3
    Join Date
    Jul 2014
    Thanked 13 Times in 12 Posts
    IMHO if you want it to be general purpose oriented it should compress slightly faster and tighter than 7z 1st of all.

  6. #4
    Join Date
    Dec 2011
    Cambridge, UK
    Thanked 186 Times in 127 Posts
    If you're focusing on things like RAM disk and byte oriented packing, then likely comparing to LZ4 and/or Snappy are the obvious existing contenders out there.

    Give those a whirl on some data and see how your one compares.
    Good luck.

Similar Threads

  1. XORpressor. Yet another idea for a compressor
    By Cristo in forum Data Compression
    Replies: 5
    Last Post: 24th August 2016, 17:29
  2. Anyone thought of this yet? (Image Compressor Idea)
    By Lucas in forum Data Compression
    Replies: 12
    Last Post: 5th May 2015, 14:48
  3. Multi-way QuickSort
    By Piotr Tarsa in forum The Off-Topic Lounge
    Replies: 7
    Last Post: 9th May 2011, 19:59
  4. pxz: Multi threaded xz compressor
    By polemon in forum Data Compression
    Replies: 0
    Last Post: 21st March 2011, 03:55
  5. Multi-threaded compression
    By Cyan in forum Data Compression
    Replies: 34
    Last Post: 16th January 2011, 18:32

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts