TurboPFor (Integer Compression) vs blosc (Binary compression)

[dnd]

First of all blosc is claiming to be faster than memcpy.
This is impossible, simply because blosc is no more
than a shuffle (or transpose) + lz77.
Only the shuffle alone cannot be faster than memcpy, because
shuffle is a blocked memcpy with zero compression.
The Shuffle is applied before compression and after decompression.


A multithread blosc muss be also compared to a multithreaded memcpy.


We can see from the benchmarks at TurboPFor that "Integer Compression" is superior to "binary compression"
like blosc in terms of compression ratio and speed.
Blosc is tested w. lz4 as compressor and vectorized shuffle.


Another advantage of Integer Compression is direct access to individual values without decompressing entire blocks.

[Piotr Tarsa]

First of all blosc is claiming to be faster than memcpy.
This is impossible, simply because blosc is no more
than a shuffle (or transpose) + lz77.
Only the shuffle alone cannot be faster than memcpy, because
shuffle is a blocked memcpy with zero compression.
The Shuffle is applied before compression and after decompression.

Why? If stream is divided into small enough blocks then all intermediate data can fit in caches so we can get faster than memcpy. Memcpy has to read uncompressed input and write uncompressed output, that's 2x input size. Codec that keeps all intermediate data within caches does less memory I/O than 2x input size as the compressed stream is smaller than uncompressed one.

A multithread blosc muss be also compared to a multithreaded memcpy.

Yep.

We can see from the benchmarks at TurboPFor that "Integer Compression" is superior to "binary compression"
like blosc in terms of compression ratio and speed.
Blosc is tested w. lz4 as compressor and vectorized shuffle.

It's possible that blosc authors can provide other benchmarks in which they win. We need independent benchmarkers to make objective opinion.

Another advantage of Integer Compression is direct access to individual values without decompressing entire blocks.

That's nice, although it also depends on use case - what if the user actually reads input data in small blocks?

[dnd]

Why? If stream is divided into small enough blocks the...

Well, as i have explained the shuffle function alone is a variant of memcpy. Copy the same amount of bytes only to different locations. Addionally you need lz77 after preprocessing.
You have also Blosc = shuffle + lz77.
This is independent from the data or caching. "shuffle + lz77" cannot be faster than memcpy, because "memcpy speed" >= "shuffle speed" as the same amount of bytes is transferred.

[Piotr Tarsa]

But the intermediate copy is done within caches. Authors didn't claim that blosc is faster than memcpy for all input sizes and input caching status. But even with small input they could achieve speed over memcpy if the input is uncached prior to compression/ decompression.

[Piotr Tarsa]

Okay. If I would say that blosc decompression is faster than system-RAM to system-RAM transfer of uncompressed data, then would that be possible? And why (or why not)?