Results 1 to 5 of 5

Thread: Optimized compression codecs on Intel's Clear Linux

  1. #1
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    353
    Thanks
    131
    Thanked 54 Times in 38 Posts

    Optimized compression codecs on Intel's Clear Linux

    Hi all – It looks like Intel's Clear Linux distro ships with optimized compression codecs. They wrote a post about it here, and you can see the graphs for how much of a difference it makes – they seem pretty small, but I'd rather have them than not have them.

    For the XZ/LZMA program, it looks like maybe the only thing they did was use O3. For zlib, I think they did more than that. Intel has an optimized zlib library here, and I think there's also one that was geared toward genetic data – it's not clear which optimized library they're using in Clear Linux.

  2. Thanks (3):

    Cyan (10th March 2017),inikep (10th March 2017),JamesB (10th March 2017)

  3. #2
    Member
    Join Date
    Feb 2013
    Location
    San Diego
    Posts
    1,057
    Thanks
    54
    Thanked 72 Times in 56 Posts
    Zstandard is getting a lot of positive buzz. Good work, Yann.

  4. #3
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    506
    Thanks
    187
    Thanked 177 Times in 120 Posts
    Nice report, but they miss out entirely the decompression speeds. This is very different and you may draw different conclusions. Eg xz -1 is better than gzip -9; maybe not if you care about rapid uncompression. Similarly the bzip2 vs xz graphs, and of course we all know zstd is very fast at decompression.

  5. #4
    Member
    Join Date
    Feb 2013
    Location
    San Diego
    Posts
    1,057
    Thanks
    54
    Thanked 72 Times in 56 Posts
    The discussion of speed/compression was not very well-done imo. It showed graphs of compressor performance without revealing the source data it was tested on. Clearly, the data being compressed has an enormous impact on speed/performance, particularly when comparing different compressors based on completely different algorithms.

    The discussion here and people's blogs are at a much higher level.

    IME, bzip2 and xz can both be slow, and sometimes xz is much slower. My sense is that xz is more variable depending on input and is prone to bad cases. bzip2 is just doing the BWT on blocks; there's only so much it can slow down, while xz/lzma depends on an enormous variety of tricks.

  6. #5
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    353
    Thanks
    131
    Thanked 54 Times in 38 Posts
    Yeah, it wasn't very rigorous wrt identifying the data and decompression performance. And I think replacing zlib with a clean-sheet effort like libdeflate would be best in the end.

    LZMA still surprises me – it's #1 in ratio on Turbo Bench right now: https://github.com/powturbo/TurboBench

    LzTurbo surprises me too – it looks like... the best general purpose compression codec on earth. If it's as good as reported in these benchmarks, I'm surprised people don't talk about it more. I'm vaguely aware of an issue Bulat has with its code use / license issues, but that wouldn't explain why no one talks about it. I wonder if there's a catch, like a huge CPU hit or something.

Similar Threads

  1. Adding new codecs to 7z.dll
    By Shelwien in forum Data Compression
    Replies: 1
    Last Post: 12th June 2016, 00:51
  2. New Compression Codecs Risk Making Zlib Obsolete
    By dnd in forum Data Compression
    Replies: 7
    Last Post: 19th January 2016, 19:01
  3. Intel IPP compression support
    By encode in forum Data Compression
    Replies: 6
    Last Post: 4th February 2014, 15:35
  4. Linux memory manager discussion, same problem as compression
    By nburns in forum The Off-Topic Lounge
    Replies: 10
    Last Post: 11th December 2013, 21:44
  5. Compression benchmarking: 64 bit images and 24 bit codecs
    By m^2 in forum The Off-Topic Lounge
    Replies: 6
    Last Post: 30th November 2011, 16:01

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •