Results 1 to 7 of 7

Thread: Intel IPP compression support

  1. #1
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,013
    Thanks
    404
    Thanked 403 Times in 153 Posts

    Lightbulb Intel IPP compression support

    Intel Integrated Performance Primitives (IPP) includes a nice set of compression-related stuff - Deflate, BZip2, LZO, LZSS, BWTFwd/BWTInv, ... Nice read:
    http://software.intel.com/sites/prod...3_0_Intro.html

  2. Thanks:

    nburns (3rd February 2014)

  3. #2
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,257
    Thanks
    307
    Thanked 797 Times in 489 Posts
    I wonder if Intel plans to extend the instruction set or add hardware optimizations to support their library.

  4. #3
    Member
    Join Date
    Jun 2008
    Location
    G
    Posts
    377
    Thanks
    26
    Thanked 23 Times in 16 Posts
    Quote Originally Posted by Matt Mahoney View Post
    I wonder if Intel plans to extend the instruction set or add hardware optimizations to support their library.
    i think that is the logical consequense, its just a matter of time, but maybe we will see in the next tock, what will happen

  5. #4
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,497
    Thanks
    26
    Thanked 132 Times in 102 Posts
    There are hardware Deflate solutions already, so Intel is late to the party :P

    http://en.wikipedia.org/wiki/DEFLATE#Hardware_encoders

    And bzip2 is already being obsoleted by lzma/ lzma2.

  6. #5
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,257
    Thanks
    307
    Thanked 797 Times in 489 Posts
    I think I recall a conversation a year or two ago about Intel asking how they could optimize compression. My response was that for high end algorithms like PPM or CM that the bottleneck is random memory access, so the best thing they could do was design faster memories and larger caches.

    But it seems to me that the most logical target would be to optimize video decompression. Maybe instructions for fast IDCT.

    Another possibility is hardware and OS support for compressed memory. For example, blocks of zero bytes don't have to be physically allocated.

  7. #6
    Member just a worm's Avatar
    Join Date
    Aug 2013
    Location
    planet "earth"
    Posts
    96
    Thanks
    29
    Thanked 6 Times in 5 Posts
    If I understood thouse "Integrated Performance Primitives" correctly then this is a bunch of libraries for Linux and Windows. I think the libraries for Windows where *.dll files which only had a few functions (they split all the stuff up so there are many *.dll files) and every *.dll file is something like 2 MB in size. The whole package for Windows is something like 400 MB. So I don't think that this library package is beeing used widely.

    Did anyone ever work with thouse libraries? How are they doing regarding speed and memory consumption?

    My response was that for high end algorithms like PPM or CM that the bottleneck is random memory access, so the best thing they could do was design faster memories and larger caches.
    Yes, that's right. If you are trying to do a BWT detransform, then you have the same problem.

    But I don't think that larger caches really fixes the problem. It helps but it rather reduces the problem to the data that still doesn't fit in the cache because of it's still to big. Usually the data you need to process is also several times bigger than the actual payload.

    One thing that intel has done in the near past was adding support to non-blocking memory accesses to the Atom-series. The Atom-series is probably the most sold processor serie at the time. Maybe this also helps a little to increase the speed. But you would probably need to do some adjustments in the code to make good use of the "out-of-order"-capabilities of a processor because it's somewhat different to an "in-order"-processor (like the older generation of the Atom-processors are).

    The most other processors already had support for non-blocking memory accesses. I don't think that AMD ever released another "in-order execution"-processor after their first "out-of-order execution"-processor.

    In the last months I tried to do some speed improvements by using the instruction set extensions. I used the SSE2 instruction set extension with the "16 byte"-XMM-registers to improve the detransformation speed of a BWT. I could only use it for some trivial tasks instead of anything serious so the speed improvement was less than 10% but it was very much work to look into all this stuff. So my personal opinion about thouse instruction set extensions is, that they are somewhat disappointing.
    Last edited by just a worm; 4th February 2014 at 13:01.

  8. #7
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    506
    Thanks
    187
    Thanked 177 Times in 120 Posts
    I tested ipp_gzip recently and while faster, it was also poorer at compression. There was still some speed up over and above the slightly lower compression (tested against zlib using lower compression levels to get a similar size), but not a particularly large one.

  9. Thanks:

    just a worm (11th February 2014)

Similar Threads

  1. Intel AVX-512, MPX and SHA Extension
    By Bulat Ziganshin in forum Download Area
    Replies: 7
    Last Post: 1st June 2015, 21:43
  2. Intel Avoton
    By Sportman in forum Data Compression
    Replies: 0
    Last Post: 4th September 2013, 01:57
  3. gzip - Intel IPP
    By M4ST3R in forum Download Area
    Replies: 5
    Last Post: 2nd June 2010, 15:09
  4. Winzip v12.0 with JPG recompression & 7z support
    By maadjordan in forum Data Compression
    Replies: 3
    Last Post: 12th September 2008, 23:58
  5. GCC mmx support
    By toffer in forum Forum Archive
    Replies: 6
    Last Post: 19th January 2008, 13:20

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •