Results 1 to 11 of 11

Thread: zstd HC

  1. #1
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    I found a HC branch of zstd:
    https://github.com/Cyan4973/zstd/tree/HC

    The ratio is much closer to brotli and decompression is 2 times faster.

    Code:
    zstd v0.2                       427 ms (239 MB/s), 51367682, 189 ms (541 MB/s)
    zstd_HC v0.2 -1                 996 ms (102 MB/s), 48918586, 244 ms (419 MB/s)
    zstd_HC v0.2 -2                 1152 ms (88 MB/s), 47193677, 247 ms (414 MB/s)
    zstd_HC v0.2 -3                 2810 ms (36 MB/s), 45401096, 249 ms (411 MB/s)
    zstd_HC v0.2 -4                 4009 ms (25 MB/s), 44631716, 243 ms (421 MB/s)
    zstd_HC v0.2 -5                 4904 ms (20 MB/s), 44529596, 244 ms (419 MB/s)
    zstd_HC v0.2 -6                 5785 ms (17 MB/s), 44281722, 244 ms (419 MB/s)
    zstd_HC v0.2 -7                 7352 ms (13 MB/s), 44108456, 241 ms (424 MB/s)
    zstd_HC v0.2 -8                 9378 ms (10 MB/s), 43978979, 237 ms (432 MB/s)
    zstd_HC v0.2 -9                 12306 ms (8 MB/s), 43901117, 237 ms (432 MB/s)
    brotli 2015-10-11 level 0       1202 ms (85 MB/s), 47882059, 489 ms (209 MB/s)
    brotli 2015-10-11 level 3       1739 ms (58 MB/s), 47451223, 475 ms (215 MB/s)
    brotli 2015-10-11 level 4       2379 ms (43 MB/s), 46943273, 477 ms (214 MB/s)
    brotli 2015-10-11 level 5       6175 ms (16 MB/s), 43363897, 528 ms (193 MB/s)
    brotli 2015-10-11 level 6       9474 ms (10 MB/s), 42877293, 463 ms (221 MB/s)
    Last edited by inikep; 28th October 2015 at 21:11.

  2. Thanks (7):

    Bulat Ziganshin (29th October 2015),Cyan (29th October 2015),Jarek (28th October 2015),Jyrki Alakuijala (29th October 2015),Stephan Busch (28th October 2015),tobijdc (28th October 2015),Turtle (29th October 2015)

  3. #2
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    773
    Thanks
    237
    Thanked 248 Times in 152 Posts
    Nice ... so LZFSE_HC in iOS10?

  4. #3
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    474
    Thanked 175 Times in 85 Posts
    Can you please provide a compiled Version?
    Win32 or win64?

  5. #4
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    I assume that Cyan will add support for zstd_HC in zstd.exe when it will be finished.

  6. #5
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    204
    Thanks
    172
    Thanked 110 Times in 66 Posts
    here's a win32 binary, no idea if that runs properly... i just compiled it.

    /edit: added an also untested x64 binary.
    Attached Files Attached Files
    Last edited by load; 29th October 2015 at 14:24. Reason: added x64

  7. Thanks:

    Stephan Busch (30th October 2015)

  8. #6
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    829
    Thanks
    239
    Thanked 300 Times in 180 Posts
    Quote Originally Posted by inikep View Post
    I found a HC branch of zstd:
    The ratio is much closer to brotli and decompression is 2 times faster.
    What happens if you run both with a 100 kB html document?

    What window size settings do you run the test with?

    Why stop the data at brotli level 6? Brotli level 11 is still practical and a cost-efficient solution for static resources. Even at quality 11, the bandwidth savings cover the CPU cost after the file has been served 5-10 times.

  9. #7
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    What happens if you run both with a 100 kB html document?

    What window size settings do you run the test with?

    Why stop the data at brotli level 6? Brotli level 11 is still practical and a cost-efficient solution for static resources. Even at quality 11, the bandwidth savings cover the CPU cost after the file has been served 5-10 times.
    I was interested in compression of gigabytes of data at 10+ MB/s. I didn't also test zstd_HC levels 10-25.

    The window size for zstd_HC levels 1-9 is 18-21 bits.

    I think brotli will be better for 100 kB html because of a static dictionary (but it can also be embedded into zstd).

  10. #8
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    885
    Thanks
    480
    Thanked 278 Times in 118 Posts
    Version 0.3 is out and feature ... drumble .... HC compression modes !

    https://github.com/Cyan4973/zstd/releases

    OK, it was no longer a secret ...

    This is only a first version, so there is a lot of room for improvements.
    But at least, it's now possible to play with different compression levels, to better fit custom usages (typically the network communication speed).
    I tried to provide some fine-tuned levels, so there are a lot of them.
    That being said, beyond -8, I believe the trade-off between cpu and compression ratio hits diminishing returns, and will deserve some upgrade before becoming interesting.
    This will be for a later version of the algorithm.
    Last edited by Cyan; 31st October 2015 at 14:48.

  11. Thanks (4):

    Bulat Ziganshin (31st October 2015),inikep (31st October 2015),Jarek (31st October 2015),tobijdc (31st October 2015)

  12. #9
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    474
    Thanked 175 Times in 85 Posts
    can anybody please provide a win64 compile of latest ZSTD 0.3.4?
    Thank you very much

  13. #10
    Member
    Join Date
    May 2009
    Location
    France
    Posts
    99
    Thanks
    13
    Thanked 75 Times in 45 Posts
    Hello,

    Zstd 0.3.4 windows (32 & 64bit) binaries
    zstd-0.3.4.7z

    AiZ

  14. Thanks (3):

    Bulat Ziganshin (7th November 2015),Cyan (7th November 2015),Stephan Busch (7th November 2015)

  15. #11
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    885
    Thanks
    480
    Thanked 278 Times in 118 Posts
    FYI, note that the 0.3.x version of the command line utility
    is not able to extract the same performance as direct API call.

    This is because the I/O routine cut files into smaller independent blocks of 512 KB,
    thus destroying long range correlations.

    This will of course be improved in future versions,
    but it requires a change of format, so this will be for 0.4.x serie.
    Last edited by Cyan; 8th November 2015 at 15:52.

  16. Thanks:

    tobijdc (8th November 2015)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •