Not sure if people here have noticed, lz4's compression and decompression speeds change dramatically at block size 64KB. Cannot figure out the reason(s). Any thoughts?
Here are some simple tests on an Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz, running Red Hat Enterprise Linux Server release 6.10 (kernel 2.6.32). Granted, it is a quite old machine. Not sure if newer machines will have the same results:
vigeland -19- <~> lz4 -b1 enwik8
1#enwik8 : 100000000 -> 57262281 (1.746), 327.9 MB/s ,3000.0 MB/s
vigeland -20- <~> lz4 -b1 -B7 enwik8
using blocks of size 4096 KB
1#enwik8 : 100000000 -> 57285873 (1.746), 333.3 MB/s ,3000.0 MB/s
vigeland -21- <~> lz4 -b1 -B6 enwik8
using blocks of size 1024 KB
1#enwik8 : 100000000 -> 57358020 (1.743), 333.3 MB/s ,3000.0 MB/s
vigeland -22- <~> lz4 -b1 -B5 enwik8
using blocks of size 256 KB
1#enwik8 : 100000000 -> 57645644 (1.735), 330.6 MB/s ,2916.7 MB/s
vigeland -23- <~> lz4 -b1 -B4 enwik8
using blocks of size 64 KB
1#enwik8 : 100000000 -> 57001007 (1.754), 285.7 MB/s ,2307.7 MB/s
As can be seen from the above results, compression ratio doesn't change much for various block sizes, but both compression and decompression speeds decrease a lot at block size 64KB.
Any convincing explanations?