What should be the speed for realtime data (de)compression?

[lz77]

What the minimum speed should be on 1 core for typical Intel/AMD CPU 3HGz RAM 1.6HGz for realtime data compression? For example 100Mb/sec. is realtime data compression or not? (I mean enwik8 or similar test files). What about realtime decompression speed?

Is (de)compression on the fly the same as realtime (de)compression? If not, then what about the minimum speed for (de)compression on the fly?

[Jyrki Alakuijala]

https://en.wikipedia.org/wiki/Real-time_computing doesn't mean fast. It means that the lowest possible speed is known and that is taken into account as a hard system requirement.

So, it could be that a real-time compressor is decompressing 100 MB / s and a non-real-time 1000 MB / s.

[Piotr Tarsa]

In other words, real-time computing is usually about optimizing the worst scenario latency (and resources utilization).

[lz77]

Thanks but how about compression on the fly? Let LZ4 in fast mode compresses ENWIK8 in 0.8 sec. and decompresses in 0.2 sec, and my code respectively in 1.2 sec./0.35 sec. Have I the right to say that my code compresses/decompresses on the fly?

[Jyrki Alakuijala]

Thanks but how about compression on the fly? Let LZ4 in fast mode compresses ENWIK8 in 0.8 sec. and decompresses in 0.2 sec, and my code respectively in 1.2 sec./0.35 sec. Have I the right to say that my code compresses/decompresses on the fly?

That's great, but on-the-fly is not much related to speed, and often on-the-fly compression/decompression is slower than full one-shot compression/decompression. On-the-fly means that the compressor or decompressor can process the stream without keeping the whole input in memory -- and the system can interleave I/O requests between the (de)compression. On-the-fly is not well-established terminology in compression, usually we use the term 'streaming' there.

Another aspect of streaming is the granularity of blocked streaming -- how much data do we need to accumulate before we can compress the first block? How much data do we need to read before we can decompress the first kilobyte? ... and why does this matter -- because browser can start fetching the component javascript etc. that can be stored in the beginning only after that data is decompressed. Often there is an asymmetry in granularity, i.e., compression has larger blocks whereas decompression can emit data at some internal higher granularity (like commands).

[Bulat Ziganshin]

this is defined by users of your library. just two examples:
1) ntfs compression: it should compress data as they goes to HDD/SSD, so the reasonable speed will be 100-200 MB/s for HDDs and much higher for SSDs
2) Google, FB and so on may use compression to reduce traffic between their sites/servers. F.e. map/reduce technology spreads data among many computers. In this case, you need speed that can saturate their network links (probably 1 GBit for an array of inexpensive servers)
3) Web server want to deflate-compress the content it generates. In this case, you need to saturate Web server network link, which is probably 1 GBit/s too

Note that usually you don't need to use single thread, there are many tasks and usually you can split data into small chunks, so aggregate perfromance of all cpu cores/sockets is considered

[smjohn1]

In high end data storage products ( boxes ) with SSDs, usually the processing unit ( buffer ) is about 4K to 8K, and one prefers compression speed to about 1GB/s ( for data write ) and much higher ( > 5GB/s ) for decompression ( for data read ). As of now, even LZ4 cannot reach that kind of speed, so compression/decompression does become one bottleneck. FPGA implementation doesn't help much as it needs extra ( two-way ) data copy between memory and FPGA. So definitely there are many things to do in improving compression speed ( with decent compression ratio ).

[xcrh]

The only "correct" answer I could imagine is: as fast as you could get it, as long as you're ok with compression ratio. If you manage to outrun others on some particular tradeoff, that's your win, obviously. From practical standpoint, say, in Linux ppl are compressing both RAM and filesystems, which are "delay sensitive" by zlib, lzo, lz4 and zstd. So I guess "real time" lies "lz4 and faster" on high speed side and "low levels of zlib and zstd" at "slow" end. Getting stuck for a second to compress 4k page would obviously hardly be considered "realtime" compression, even if you promise never exceed one second, so it formally qualifies (in terms of guarantees).