zstd HC

[inikep]

I found a HC branch of zstd:
https://github.com/Cyan4973/zstd/tree/HC

The ratio is much closer to brotli and decompression is 2 times faster.

zstd v0.2                       427 ms (239 MB/s), 51367682, 189 ms (541 MB/s)
zstd_HC v0.2 -1                 996 ms (102 MB/s), 48918586, 244 ms (419 MB/s)
zstd_HC v0.2 -2                 1152 ms (88 MB/s), 47193677, 247 ms (414 MB/s)
zstd_HC v0.2 -3                 2810 ms (36 MB/s), 45401096, 249 ms (411 MB/s)
zstd_HC v0.2 -4                 4009 ms (25 MB/s), 44631716, 243 ms (421 MB/s)
zstd_HC v0.2 -5                 4904 ms (20 MB/s), 44529596, 244 ms (419 MB/s)
zstd_HC v0.2 -6                 5785 ms (17 MB/s), 44281722, 244 ms (419 MB/s)
zstd_HC v0.2 -7                 7352 ms (13 MB/s), 44108456, 241 ms (424 MB/s)
zstd_HC v0.2 -8                 9378 ms (10 MB/s), 43978979, 237 ms (432 MB/s)
zstd_HC v0.2 -9                 12306 ms (8 MB/s), 43901117, 237 ms (432 MB/s)
brotli 2015-10-11 level 0       1202 ms (85 MB/s), 47882059, 489 ms (209 MB/s)
brotli 2015-10-11 level 3       1739 ms (58 MB/s), 47451223, 475 ms (215 MB/s)
brotli 2015-10-11 level 4       2379 ms (43 MB/s), 46943273, 477 ms (214 MB/s)
brotli 2015-10-11 level 5       6175 ms (16 MB/s), 43363897, 528 ms (193 MB/s)
brotli 2015-10-11 level 6       9474 ms (10 MB/s), 42877293, 463 ms (221 MB/s)

[Jarek]

Nice ... so LZFSE_HC in iOS10?

[Stephan Busch]

Can you please provide a compiled Version?
Win32 or win64?

[inikep]

I assume that Cyan will add support for zstd_HC in zstd.exe when it will be finished.

[load]

here's a win32 binary, no idea if that runs properly... i just compiled it.

/edit: added an also untested x64 binary.

[Jyrki Alakuijala]

I found a HC branch of zstd:
The ratio is much closer to brotli and decompression is 2 times faster.

What happens if you run both with a 100 kB html document?

What window size settings do you run the test with?

Why stop the data at brotli level 6? Brotli level 11 is still practical and a cost-efficient solution for static resources. Even at quality 11, the bandwidth savings cover the CPU cost after the file has been served 5-10 times.

[inikep]

What happens if you run both with a 100 kB html document?

What window size settings do you run the test with?

Why stop the data at brotli level 6? Brotli level 11 is still practical and a cost-efficient solution for static resources. Even at quality 11, the bandwidth savings cover the CPU cost after the file has been served 5-10 times.

I was interested in compression of gigabytes of data at 10+ MB/s. I didn't also test zstd_HC levels 10-25.

The window size for zstd_HC levels 1-9 is 18-21 bits.

I think brotli will be better for 100 kB html because of a static dictionary (but it can also be embedded into zstd).

[Cyan]

Version 0.3 is out and feature ... drumble .... HC compression modes !

https://github.com/Cyan4973/zstd/releases

OK, it was no longer a secret ...

This is only a first version, so there is a lot of room for improvements.
But at least, it's now possible to play with different compression levels, to better fit custom usages (typically the network communication speed).
I tried to provide some fine-tuned levels, so there are a lot of them.
That being said, beyond -8, I believe the trade-off between cpu and compression ratio hits diminishing returns, and will deserve some upgrade before becoming interesting.
This will be for a later version of the algorithm.

[Stephan Busch]

can anybody please provide a win64 compile of latest ZSTD 0.3.4?
Thank you very much

[AiZ]

Hello,

Zstd 0.3.4 windows (32 & 64bit) binaries
zstd-0.3.4.7z

AiZ

[Cyan]

FYI, note that the 0.3.x version of the command line utility
is not able to extract the same performance as direct API call.

This is because the I/O routine cut files into smaller independent blocks of 512 KB,
thus destroying long range correlations.

This will of course be improved in future versions,
but it requires a change of format, so this will be for 0.4.x serie.