Results 1 to 2 of 2

Thread: Noob question : algorithms versus multi-threading

  1. #1
    Join Date
    May 2009
    Thanked 75 Times in 45 Posts

    First, please forgive me if my questions are useless or have already being discussed.

    I recently discovered this very interesting forum and its wonderful programs and hackers. As my topic says, I'm just a user, tester perhaps, of these tools, finding their performances astonishing, especially CCM for compression and QuickLZ/LZPM for speed.

    My (simple) questions are : are the algorithms behind all these programs multi-thread/multi-core adaptable? If so, can we see big improvements due to multi-threading in the near future?

    Why do I ask these ? Because, here at work, we recently receive a new computer, dedicated to compile our software. Quite a monster, even for our needs, it's a quad Opteron 8220SE, 8GB of RAM, running RHEL4. Using pbzip2 with default settings, compression of enwik8 takes less than 4 seconds! Less than 50 seconds for enwik9!

    I don't think that QuickLZ, THOR, LZPM, etc can go faster, limited by I/Os, but having CCM or QUAD running at today's QuickLZ speed for example... as it seems that future is massively multi-core oriented.

    I don't want to bother you any longer. Have a nice day!


    Edit: words missing.

  2. #2
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Thanked 698 Times in 378 Posts
    it may be rather easy accomplished by splitting data into several chunks according to number of cpus and compressing them independently. to be exact, pbzip2 does the same. its algorithm anyway can't process data in more than 900 kb chunks (limit imposed by bwd compatibility) while other compression algorithms may get some compression penalty but not so much. quad have 16mb block, i already suggested encode to make its MT variant which will compress faster than decompress

Similar Threads

  1. identity of obscure algorithms
    By asmodean in forum Data Compression
    Replies: 2
    Last Post: 6th August 2009, 09:50
  2. Noob question about dictionary size (and about rep)
    By SvenBent in forum Data Compression
    Replies: 1
    Last Post: 23rd January 2009, 01:35
  3. Multi-threading motivation
    By Trixter in forum Data Compression
    Replies: 1
    Last Post: 10th September 2008, 06:18
  4. Searching fast decompressable algorithms
    By Mimos in forum Data Compression
    Replies: 8
    Last Post: 25th July 2008, 00:58
  5. Replies: 1
    Last Post: 14th October 2007, 15:10

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts