Results 1 to 8 of 8

Thread: What should be the speed for realtime data (de)compression?

  1. #1
    Member lz77's Avatar
    Join Date
    Jan 2016
    Location
    Russia
    Posts
    60
    Thanks
    16
    Thanked 12 Times in 8 Posts

    Question What should be the speed for realtime data (de)compression?

    What the minimum speed should be on 1 core for typical Intel/AMD CPU 3HGz RAM 1.6HGz for realtime data compression? For example 100Mb/sec. is realtime data compression or not? (I mean enwik8 or similar test files). What about realtime decompression speed?

    Is (de)compression on the fly the same as realtime (de)compression? If not, then what about the minimum speed for (de)compression on the fly?

  2. #2
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    807
    Thanks
    233
    Thanked 291 Times in 173 Posts
    https://en.wikipedia.org/wiki/Real-time_computing doesn't mean fast. It means that the lowest possible speed is known and that is taken into account as a hard system requirement.

    So, it could be that a real-time compressor is decompressing 100 MB / s and a non-real-time 1000 MB / s.

  3. #3
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,488
    Thanks
    26
    Thanked 130 Times in 100 Posts
    In other words, real-time computing is usually about optimizing the worst scenario latency (and resources utilization).

  4. #4
    Member lz77's Avatar
    Join Date
    Jan 2016
    Location
    Russia
    Posts
    60
    Thanks
    16
    Thanked 12 Times in 8 Posts
    Thanks but how about compression on the fly? Let LZ4 in fast mode compresses ENWIK8 in 0.8 sec. and decompresses in 0.2 sec, and my code respectively in 1.2 sec./0.35 sec. Have I the right to say that my code compresses/decompresses on the fly?

  5. #5
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    807
    Thanks
    233
    Thanked 291 Times in 173 Posts
    Quote Originally Posted by lz77 View Post
    Thanks but how about compression on the fly? Let LZ4 in fast mode compresses ENWIK8 in 0.8 sec. and decompresses in 0.2 sec, and my code respectively in 1.2 sec./0.35 sec. Have I the right to say that my code compresses/decompresses on the fly?
    That's great, but on-the-fly is not much related to speed, and often on-the-fly compression/decompression is slower than full one-shot compression/decompression. On-the-fly means that the compressor or decompressor can process the stream without keeping the whole input in memory -- and the system can interleave I/O requests between the (de)compression. On-the-fly is not well-established terminology in compression, usually we use the term 'streaming' there.

    Another aspect of streaming is the granularity of blocked streaming -- how much data do we need to accumulate before we can compress the first block? How much data do we need to read before we can decompress the first kilobyte? ... and why does this matter -- because browser can start fetching the component javascript etc. that can be stored in the beginning only after that data is decompressed. Often there is an asymmetry in granularity, i.e., compression has larger blocks whereas decompression can emit data at some internal higher granularity (like commands).

  6. Thanks:

    lz77 (22nd November 2017)

  7. #6
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,552
    Thanks
    767
    Thanked 684 Times in 370 Posts
    this is defined by users of your library. just two examples:
    1) ntfs compression: it should compress data as they goes to HDD/SSD, so the reasonable speed will be 100-200 MB/s for HDDs and much higher for SSDs
    2) Google, FB and so on may use compression to reduce traffic between their sites/servers. F.e. map/reduce technology spreads data among many computers. In this case, you need speed that can saturate their network links (probably 1 GBit for an array of inexpensive servers)
    3) Web server want to deflate-compress the content it generates. In this case, you need to saturate Web server network link, which is probably 1 GBit/s too

    Note that usually you don't need to use single thread, there are many tasks and usually you can split data into small chunks, so aggregate perfromance of all cpu cores/sockets is considered

  8. Thanks:

    lz77 (27th November 2017)

  9. #7
    Member
    Join Date
    Feb 2016
    Location
    USA
    Posts
    91
    Thanks
    35
    Thanked 8 Times in 8 Posts
    In high end data storage products ( boxes ) with SSDs, usually the processing unit ( buffer ) is about 4K to 8K, and one prefers compression speed to about 1GB/s ( for data write ) and much higher ( > 5GB/s ) for decompression ( for data read ). As of now, even LZ4 cannot reach that kind of speed, so compression/decompression does become one bottleneck. FPGA implementation doesn't help much as it needs extra ( two-way ) data copy between memory and FPGA. So definitely there are many things to do in improving compression speed ( with decent compression ratio ).
    Last edited by smjohn1; 24th November 2017 at 22:14.

  10. Thanks:

    lz77 (27th November 2017)

  11. #8
    Member
    Join Date
    Nov 2015
    Location
    boot ROM
    Posts
    95
    Thanks
    27
    Thanked 17 Times in 15 Posts
    The only "correct" answer I could imagine is: as fast as you could get it, as long as you're ok with compression ratio. If you manage to outrun others on some particular tradeoff, that's your win, obviously. From practical standpoint, say, in Linux ppl are compressing both RAM and filesystems, which are "delay sensitive" by zlib, lzo, lz4 and zstd. So I guess "real time" lies "lz4 and faster" on high speed side and "low levels of zlib and zstd" at "slow" end. Getting stuck for a second to compress 4k page would obviously hardly be considered "realtime" compression, even if you promise never exceed one second, so it formally qualifies (in terms of guarantees).

  12. Thanks:

    lz77 (27th November 2017)

Similar Threads

  1. [LZ] M/T & GPU (de)compression
    By Bulat Ziganshin in forum Data Compression
    Replies: 0
    Last Post: 31st December 2015, 12:09
  2. Memory Usage vs. Memory Requirements for (De)Compression?
    By comp1 in forum The Off-Topic Lounge
    Replies: 7
    Last Post: 1st June 2015, 04:53
  3. compression speed VS decomp speed: which is more important?
    By Lone_Wolf236 in forum Data Compression
    Replies: 14
    Last Post: 12th July 2010, 19:57
  4. Replies: 8
    Last Post: 12th April 2009, 02:39
  5. Can't allocate memory required for (de)compression..help!
    By Duarte in forum Data Compression
    Replies: 19
    Last Post: 18th July 2008, 18:14

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •