Page 14 of 15 FirstFirst ... 412131415 LastLast
Results 391 to 420 of 438

Thread: Zstandard

  1. #391
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    937
    Thanks
    95
    Thanked 363 Times in 252 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    Note that m/t also affects level 21
    Confirmed:

    2,136,340,302 bytes, 4,604.696 sec. - 15.121 sec., zstd -21 --ultra --single-thread (v1.4.4)

  2. #392
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    883
    Thanks
    478
    Thanked 277 Times in 117 Posts
    Thanks @Sportman for exposing this use case thanks to your new benchmark !
    It made it possible to identify the issue, and @terrelln basically nailed it in a new patch.
    This issue will be fixed in the next release (v1.4.5).

  3. Thanks:

    Sportman (18th January 2020)

  4. #393
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    937
    Thanks
    95
    Thanked 363 Times in 252 Posts
    2,337,506,087 bytes, 2,596.459 sec. - 13.159 sec., zstd -18 --single-thread (v1.4.4)
    2,299,225,559 bytes, 3,196.329 sec. - 13.559 sec., zstd -19 --single-thread (v1.4.4)
    2,197,171,322 bytes, 3,947.477 sec. - 14.471 sec., zstd -20 --ultra --single-thread (v1.4.4)
    Last edited by Sportman; 18th January 2020 at 04:01.

  5. Thanks:

    Cyan (18th January 2020)

  6. #394
    Member
    Join Date
    May 2017
    Location
    United States
    Posts
    10
    Thanks
    3
    Thanked 6 Times in 4 Posts
    Thanks for the benchmarks sportman! The multithreading issue is fixed in the latest dev branch, and we now mention --single-thread in the help.

  7. Thanks (2):

    JamesB (20th January 2020),Sportman (18th January 2020)

  8. #395
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    937
    Thanks
    95
    Thanked 363 Times in 252 Posts
    Cyan and Terrelln thanks for picking up and fixing this issue so fast, can't wait till the official release to see if there are more files with till 10% storage and till 8% compression speed savings in big data production environments with zstd ultra modes.

  9. #396
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    280
    Thanks
    109
    Thanked 51 Times in 35 Posts
    I'm seeing rumblings about browser support for Zstd. What's the status?

    There's an opportunity here to achieve much better compression of HTML, CSS, JS, and SVG content. I mean better than brotli and of course gzip. The opportunity lies in building a great dictionary. None of the benchmarks I've seen for Zstd employ a dictionary. Zstd has an excellent dictionary feature that could be leverage to create a web standard static dictionary.

    Brotli has a static dictionary, but it isn't a great one. The strings are too short, it doesn't support modern HTML and JS features/keywords (because it was generated off of old HTML files), and it has a lot of strange entries, like "pittsburgh" (but not for example "Los Angeles", which is much more common) and "CIA World Factbook", which is an extremely rare string. The biggest opportunity with Zstd is to create a static dictionary that would standardize certain conventions and strings in HTML, CSS, and JS source (especially HTML). It could be developed in conjunction with a web minification standard, which we've long needed. A dictionary could then guide and optimize content creation, CMSes, and so forth. For example, it could standardize strings like the beginning of an HTML file as
    Code:
    <!DOCTYPE html><html lang=
    CMSes and minifiers could then implement the standardized strings and minification conventions. There could be hundreds of such standardized strings...

    Anyone know if a web dictionary is already being worked on?

  10. Thanks:

    Cyan (2nd April 2020)

  11. #397
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    812
    Thanks
    234
    Thanked 292 Times in 174 Posts
    Quote Originally Posted by SolidComp View Post
    I'm seeing rumblings about browser support for Zstd. What's the status?

    There's an opportunity here to achieve much better compression of HTML, CSS, JS, and SVG content. I mean better than brotli and of course gzip.
    I'm not aware of a single file of HTML, CSS, JS or SVG where Zstd compresses more than brotli. Even if brotli's internal static dictionary is disabled, brotli still wins these tests by 5 % in density.

    Quote Originally Posted by SolidComp View Post
    The opportunity lies in building a great dictionary. None of the benchmarks I've seen for Zstd employ a dictionary. Zstd has an excellent dictionary feature that could be leverage to create a web standard static dictionary.
    The Zstd dictionaries can be used with brotli, too.

    Shared brotli supports a more flexible custom dictionary format than either Zstd or normal Brotli, where the word transforms are available .

    Quote Originally Posted by SolidComp View Post
    Brotli has a static dictionary, but it isn't a great one.
    It is better than any dictionary I have seen so far.

    While people can read the dictionary and speculate, no one has actually made an experiment that shows better performance with a similarly sized dictionary on general purpose web benchmark. I suspect it is very difficult to do and certainly impossible with a simple LZ custom dictionary such as zstd's. Brotli's word based dictionary saves about 2 bits per word reference due to the more compact representation by addressing words instead of being able to address between of words or combinations of consequent words or combinations of word fragments.

    We can achieve 50 % more compression for specific use cases, but the web didn't change that much that the brotli dictionary would need an update. Actually, the brotli dictionary is well suited for compressing human communications from 200 years ago and will likely work in another 200 years.

    Quote Originally Posted by SolidComp View Post
    Anyone know if a web dictionary is already being worked on?
    https://datatracker.ietf.org/doc/dra...brotli-format/

  12. Thanks:

    SolidComp (3rd April 2020)

  13. #398
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    883
    Thanks
    478
    Thanked 277 Times in 117 Posts
    > Zstd has an excellent dictionary feature that could be leverage to create a web standard static dictionary.

    We do agree.
    We measured fairly substantial compression ratio improvements by using this technique and applying it to general websites,
    achieving much better compression ratio than any other technique available today, using a small set of static dictionaries.
    (Gains are even better when using site-dedicated dictionaries, but that's a different topic.)



    > Anyone know if a web dictionary is already being worked on?

    Investing time on this topic only makes sense if at least one significant browser manufacturer is willing to ship it. This is a very short list.

    Hence we discussed the topic with Mozilla, even going as far as inviting them to control the decision process regarding dictionaries' content.
    One can follow it here : https://github.com/mozilla/standards...ons/issues/105


    Since this strategy failed, maybe working in the other direction would make better sense :
    produce an initial set of static dictionaries, publish them as candidates, demonstrate the benefits with clear measurements,
    then try to get browsers on board, possibly create a fork as a substantial demo.

    The main issue is that it's a lot of work.
    Who has enough time to invest in this direction, knowing that even if benefits are clearly and unquestionably established,
    one should be prepared to see this work dismissed with hand-waved comments because "not invented here" ?

    So I guess the team investing time on this topic should better have strong relations with relevant parties,
    such as Mozilla, Chrome and of course the Web Committee.
    Last edited by Cyan; 2nd April 2020 at 09:02.

  14. Thanks (3):

    Jarek (3rd April 2020),Mike (2nd April 2020),SolidComp (3rd April 2020)

  15. #399
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    280
    Thanks
    109
    Thanked 51 Times in 35 Posts
    Jyrki, everything you said is correct, but you're missing something. One of my core ideas here is a prospective dictionary. Your dictionary is retrospective — it was generated based on old or existing HTML source. As a result, it doesn't support modern and near future HTML source very well.

    A prospective dictionary has two advantages:


    1. Efficient encoding of modern and foreseeable near future syntax.
    2. The ability to shape and influence syntax conventions as publishers optimize for the dictionary – a feedback loop.


    In reality a prospective dictionary of the sort I'm advocating would be a combination of prospective and retrospective.

    The tests you're asking for are nearly impossible since by definition a prospective dictionary isn't based on existing source files, but rather is intended to standardize and shape future source files. The kind of syntax a good prospective dictionary could include are things like:

    Code:
    
    <meta name="viewport" content="width=device-width"> (an increasingly common bit of code)
    <link rel="dns-prefetch" href="https://
    <link rel="preconnect" href="https://
    <link crossorigin="anonymous" media="all" rel="stylesheet" href="https://
    


    These are all nice and chunky strings, and they all represent syntax that could be standardized for the next decade or so.
    If they were present in the brotli or Zstd dictionary, publishers would then optimize their source to include these strings in this exact standardized form.
    What I mean is having the rel before the href and so forth.
    Note that these are just a few examples. A dictionary built this way would be a lot more efficient than the status quo.



  16. #400
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    812
    Thanks
    234
    Thanked 292 Times in 174 Posts
    Quote Originally Posted by SolidComp View Post
    As a result, it doesn't support modern and near future HTML source very well. ... A dictionary built this way would be a lot more efficient than the status quo.
    Perhaps you are more focused on aesthetics and elegance than efficiency. Efficiency is something that can be measured in a benchmark, not by reasoning. As an example when I played with dictionary generation (both zstd --train and shared brotli), occasionally I found that taking 10 random samples of the data and finding the best sample as a dictionary turned out more efficient than running either of the more expensive dictionary extraction algorithm. Other times concatenating 10 random samples was a decent strategy. It is not necessary for thorough thinking, logic and beauty to 'win' the dictionary efficiency game.

    Depending on how well the integration of a shared dictionary has been done, different 'rotting' times can be observed. SDCH dictionaries were rotting every 3-6 months into being mostly useless or already slightly harmful, with brotli dictionaries we barely see rot at all. Zstd dictionaries use -- while less efficient than shared brotli style shared dictionary coding -- also likely rots much slower than SDCH dictionaries. This is because SDCH used to mix the entropy originating from the dictionary use with the literals in the data, and then hope that a general purpose compressor can make sense out of this bit porridge.

    IMHO, we could come up with a unified way to do shared dictionaries and use it across the simple compressors (like zstd and brotli).

  17. #401
    Member
    Join Date
    Jun 2018
    Location
    Yugoslavia
    Posts
    53
    Thanks
    8
    Thanked 2 Times in 2 Posts
    You might then move to some entirely new format, more efficient than html and compress that.

    Does brotli pack each file separately? If so, huge gains could be made if all text files for that page were solid packed in one. iirc, time is mostly wasted on fetching multitude of small files.

  18. #402
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    812
    Thanks
    234
    Thanked 292 Times in 174 Posts
    Agreed. Html and xml are not great for transport and parsing efficiency. Packing things together can become dangerous due to enabling new attacks.

  19. #403
    Member
    Join Date
    Jan 2015
    Location
    Hungary
    Posts
    12
    Thanks
    21
    Thanked 7 Times in 3 Posts
    Report : not equal (deterministic) result.

    -T2 and -T3 Ok. -T4 level 17,18,19 different. In the meantime, better testing in the second part.

    amd.tar = ​Win7-64Bit-Radeon-Software-Adrenalin-2019-Edition-19.8.1-Aug12 (After installation (unpacking) from the C:\AMD folder.)

    System : i5-6500 , 8 GB RAM .

    Code:
    C:\m3>zstd144 -T2 -b1 -e19 amd.tar
     1#amd.tar           : 800236544 -> 667534580 (1.199),1121.8 MB/s ,1169.4 MB/s
     2#amd.tar           : 800236544 -> 657580488 (1.217), 960.1 MB/s ,1143.0 MB/s
     3#amd.tar           : 800236544 -> 649905062 (1.231), 625.7 MB/s ,1128.8 MB/s
     4#amd.tar           : 800236544 -> 645986907 (1.239), 315.0 MB/s ,1113.8 MB/s
     5#amd.tar           : 800236544 -> 640200170 (1.250),  85.4 MB/s ,1087.8 MB/s
     6#amd.tar           : 800236544 -> 639032443 (1.252),  59.2 MB/s ,1086.5 MB/s
     7#amd.tar           : 800236544 -> 637594809 (1.255),  57.1 MB/s ,1110.8 MB/s
     8#amd.tar           : 800236544 -> 637038557 (1.256),  54.1 MB/s ,1100.4 MB/s
     9#amd.tar           : 800236544 -> 636284012 (1.258),  50.3 MB/s ,1112.1 MB/s
    10#amd.tar           : 800236544 -> 633831768 (1.263),  39.7 MB/s ,1094.3 MB/s
    11#amd.tar           : 800236544 -> 633269463 (1.264),  37.9 MB/s ,1084.6 MB/s
    12#amd.tar           : 800236544 -> 633100095 (1.264),  35.5 MB/s ,1083.7 MB/s
    13#amd.tar           : 800236544 -> 633268488 (1.264),  27.4 MB/s ,1095.3 MB/s
    14#amd.tar           : 800236544 -> 632816874 (1.265),  26.6 MB/s ,1082.2 MB/s
    15#amd.tar           : 800236544 -> 632538259 (1.265),  22.8 MB/s ,1087.1 MB/s
    16#amd.tar           : 800236544 -> 630478557 (1.269),  14.7 MB/s ,1072.8 MB/s
    17#amd.tar           : 800236544 -> 625562609 (1.279),  10.5 MB/s , 985.1 MB/s
    18#amd.tar           : 800236544 -> 620804875 (1.289),  9.19 MB/s , 834.9 MB/s
    19#amd.tar           : 800236544 -> 620331125 (1.290),  7.64 MB/s , 822.5 MB/s
    
    C:\m3>zstd144 -T3 -b1 -e19 amd.tar
     1#amd.tar           : 800236544 -> 667534580 (1.199),1564.0 MB/s ,1160.3 MB/s
     2#amd.tar           : 800236544 -> 657580488 (1.217),1344.7 MB/s ,1140.4 MB/s
     3#amd.tar           : 800236544 -> 649905062 (1.231), 778.8 MB/s ,1132.0 MB/s
     4#amd.tar           : 800236544 -> 645986907 (1.239), 298.0 MB/s ,1113.3 MB/s
     5#amd.tar           : 800236544 -> 640200170 (1.250),  92.9 MB/s ,1085.9 MB/s
     6#amd.tar           : 800236544 -> 639032443 (1.252),  69.7 MB/s ,1087.1 MB/s
     7#amd.tar           : 800236544 -> 637594809 (1.255),  66.9 MB/s ,1100.0 MB/s
     8#amd.tar           : 800236544 -> 637038557 (1.256),  63.6 MB/s ,1114.8 MB/s
     9#amd.tar           : 800236544 -> 636284012 (1.258),  62.6 MB/s ,1100.0 MB/s
    10#amd.tar           : 800236544 -> 633831768 (1.263),  51.4 MB/s ,1089.4 MB/s
    11#amd.tar           : 800236544 -> 633269463 (1.264),  48.2 MB/s ,1081.3 MB/s
    12#amd.tar           : 800236544 -> 633100095 (1.264),  45.7 MB/s ,1076.5 MB/s
    13#amd.tar           : 800236544 -> 633268488 (1.264),  37.0 MB/s ,1089.3 MB/s
    14#amd.tar           : 800236544 -> 632816874 (1.265),  37.2 MB/s ,1087.7 MB/s
    15#amd.tar           : 800236544 -> 632538259 (1.265),  31.3 MB/s ,1083.6 MB/s
    16#amd.tar           : 800236544 -> 630478557 (1.269),  20.0 MB/s ,1080.3 MB/s
    17#amd.tar           : 800236544 -> 625562609 (1.279),  14.9 MB/s , 988.3 MB/s
    18#amd.tar           : 800236544 -> 620804875 (1.289),  12.8 MB/s , 847.2 MB/s
    19#amd.tar           : 800236544 -> 620331125 (1.290),  10.7 MB/s , 824.9 MB/s
    
    C:\m3>zstd144 -T4 -b1 -e19 amd.tar
     1#amd.tar           : 800236544 -> 667534580 (1.199),1959.8 MB/s ,1163.4 MB/s
     2#amd.tar           : 800236544 -> 657580488 (1.217),1663.0 MB/s ,1135.7 MB/s
     3#amd.tar           : 800236544 -> 649905062 (1.231), 839.8 MB/s ,1134.6 MB/s
     4#amd.tar           : 800236544 -> 645986907 (1.239), 254.1 MB/s ,1115.1 MB/s
     5#amd.tar           : 800236544 -> 640200170 (1.250), 103.4 MB/s ,1086.8 MB/s
     6#amd.tar           : 800236544 -> 639032443 (1.252),  79.6 MB/s ,1080.2 MB/s
     7#amd.tar           : 800236544 -> 637594809 (1.255),  72.9 MB/s ,1049.2 MB/s
     8#amd.tar           : 800236544 -> 637038557 (1.256),  72.5 MB/s ,1091.8 MB/s
     9#amd.tar           : 800236544 -> 636284012 (1.258),  71.4 MB/s ,1110.2 MB/s
    10#amd.tar           : 800236544 -> 633831768 (1.263),  59.7 MB/s ,1038.3 MB/s
    11#amd.tar           : 800236544 -> 633269463 (1.264),  59.2 MB/s ,1088.9 MB/s
    12#amd.tar           : 800236544 -> 633100095 (1.264),  56.1 MB/s ,1087.6 MB/s
    13#amd.tar           : 800236544 -> 633268488 (1.264),  45.9 MB/s ,1092.6 MB/s
    14#amd.tar           : 800236544 -> 632816874 (1.265),  45.6 MB/s ,1083.5 MB/s
    15#amd.tar           : 800236544 -> 632538259 (1.265),  39.5 MB/s ,1086.0 MB/s
    16#amd.tar           : 800236544 -> 630478557 (1.269),  23.8 MB/s , 989.1 MB/s
    17#amd.tar           : 800236544 -> 625564014 (1.279),  17.4 MB/s , 984.7 MB/s
    18#amd.tar           : 800236544 -> 620817671 (1.289),  14.9 MB/s , 836.3 MB/s
    19#amd.tar           : 800236544 -> 620342218 (1.290),  12.0 MB/s , 823.2 MB/s
    
    inspection :
    
    C:\m3>zstd144 -T4 -b17 -e19 amd.tar
    17#amd.tar           : 800236544 -> 625564014 (1.279),  18.2 MB/s , 978.3 MB/s
    18#amd.tar           : 800236544 -> 620817671 (1.289),  15.0 MB/s , 842.3 MB/s
    19#amd.tar           : 800236544 -> 620342218 (1.290),  12.2 MB/s , 826.5 MB/s
    
    C:\m3>zstd144 -T3 -b17 -e19 amd.tar
    17#amd.tar           : 800236544 -> 625562609 (1.279),  14.9 MB/s ,1000.0 MB/s
    18#amd.tar           : 800236544 -> 620804875 (1.289),  12.8 MB/s , 772.3 MB/s
    19#amd.tar           : 800236544 -> 620331125 (1.290),  10.6 MB/s , 820.7 MB/s
    
    enwik8 OK
    
    C:\m3>zstd144 -T4 -b17 -e19 enwik8
    17#enwik8            : 100000000 ->  27715412 (3.608),  5.52 MB/s , 687.4 MB/s
    18#enwik8            : 100000000 ->  27341168 (3.657),  4.92 MB/s , 614.4 MB/s
    19#enwik8            : 100000000 ->  26955340 (3.710),  4.17 MB/s , 551.2 MB/s
    
    C:\m3>zstd144 -T3 -b17 -e19 enwik8
    17#enwik8            : 100000000 ->  27715412 (3.608),  5.66 MB/s , 717.2 MB/s
    18#enwik8            : 100000000 ->  27341168 (3.657),  5.03 MB/s , 633.7 MB/s
    19#enwik8            : 100000000 ->  26955340 (3.710),  4.18 MB/s , 560.6 MB/s
    
    HWBOT X265 2.2 cpu-z 1.89.1    different
    
    C:\m3>zstd144 -T4 -b17 -e19 HWBOT.tar
    17#HWBOT.tar         : 661886464 -> 502253200 (1.318),  17.2 MB/s ,2114.9 MB/s
    18#HWBOT.tar         : 661886464 -> 500205136 (1.323),  13.7 MB/s ,1791.9 MB/s
    19#HWBOT.tar         : 661886464 -> 499767355 (1.324),  11.1 MB/s ,1826.6 MB/s
    
    C:\m3>zstd144 -T3 -b17 -e19 HWBOT.tar
    17#HWBOT.tar         : 661886464 -> 502243623 (1.318),  14.3 MB/s ,2118.5 MB/s
    18#HWBOT.tar         : 661886464 -> 500188966 (1.323),  12.2 MB/s ,1865.1 MB/s
    19#HWBOT.tar         : 661886464 -> 499762870 (1.324),  9.85 MB/s ,1826.1 MB/s
    Second part : Movie (H264/AAC) and CinebenchR20.060 more differences. Marking the different values in the last column. Level 22 no testing 8 GB RAM only 2 threads run.

    Code:
    Movie
    
    ​C:\m3>zstd144 -T2 -b1 -e21 movie.mkv
     1#movie.mkv         :1186646289 ->1186603658 (1.000),1965.6 MB/s ,6958.7 MB/s
     2#movie.mkv         :1186646289 ->1186602965 (1.000),1976.4 MB/s ,7076.6 MB/s
     3#movie.mkv         :1186646289 ->1186594049 (1.000),1675.4 MB/s ,6966.2 MB/s
     4#movie.mkv         :1186646289 ->1186592964 (1.000),1444.6 MB/s ,6908.6 MB/s
     5#movie.mkv         :1186646289 ->1186579688 (1.000), 237.5 MB/s ,6974.9 MB/s
     6#movie.mkv         :1186646289 ->1186570397 (1.000), 186.9 MB/s ,6770.0 MB/s
     7#movie.mkv         :1186646289 ->1186563987 (1.000), 181.1 MB/s ,6767.4 MB/s
     8#movie.mkv         :1186646289 ->1186563386 (1.000), 129.4 MB/s ,6210.1 MB/s
     9#movie.mkv         :1186646289 ->1186558263 (1.000), 106.3 MB/s ,5760.3 MB/s
    10#movie.mkv         :1186646289 ->1186546359 (1.000),  77.0 MB/s ,6352.7 MB/s
    11#movie.mkv         :1186646289 ->1186542815 (1.000),  66.9 MB/s ,6257.6 MB/s
    12#movie.mkv         :1186646289 ->1186542565 (1.000),  70.3 MB/s ,6667.2 MB/s
    13#movie.mkv         :1186646289 ->1186543214 (1.000),  61.3 MB/s ,6915.3 MB/s
    14#movie.mkv         :1186646289 ->1186542413 (1.000),  53.5 MB/s ,6611.0 MB/s
    15#movie.mkv         :1186646289 ->1186539987 (1.000),  51.6 MB/s ,6129.9 MB/s
    16#movie.mkv         :1186646289 ->1186516228 (1.000),  14.6 MB/s ,5927.3 MB/s
    17#movie.mkv         :1186646289 ->1186494618 (1.000), 10.17 MB/s ,6259.7 MB/s
    18#movie.mkv         :1186646289 ->1186421222 (1.000),  9.07 MB/s ,6054.9 MB/s
    19#movie.mkv         :1186646289 ->1186033076 (1.001),  7.50 MB/s ,5397.7 MB/s
    20#movie.mkv         :1186646289 ->1186167024 (1.000),  5.71 MB/s ,6323.6 MB/s
    21#movie.mkv         :1186646289 ->1186255066 (1.000),  5.20 MB/s ,5947.8 MB/s
    
    C:\m3>zstd144 -T3 -b1 -e21 movie.mkv
     1#movie.mkv         :1186646289 ->1186604884 (1.000),2579.8 MB/s ,6968.6 MB/s    < 1
     2#movie.mkv         :1186646289 ->1186602965 (1.000),2558.1 MB/s ,7008.9 MB/s
     3#movie.mkv         :1186646289 ->1186594049 (1.000),2109.8 MB/s ,6946.4 MB/s
     4#movie.mkv         :1186646289 ->1186592964 (1.000),1690.0 MB/s ,7039.7 MB/s
     5#movie.mkv         :1186646289 ->1186579688 (1.000), 228.0 MB/s ,6918.0 MB/s
     6#movie.mkv         :1186646289 ->1186570397 (1.000), 193.1 MB/s ,6880.9 MB/s
     7#movie.mkv         :1186646289 ->1186563987 (1.000), 190.6 MB/s ,6981.5 MB/s
     8#movie.mkv         :1186646289 ->1186563386 (1.000), 195.5 MB/s ,6962.2 MB/s
     9#movie.mkv         :1186646289 ->1186558263 (1.000), 131.6 MB/s ,6887.4 MB/s
    10#movie.mkv         :1186646289 ->1186546359 (1.000),  97.1 MB/s ,6935.3 MB/s
    11#movie.mkv         :1186646289 ->1186542815 (1.000),  82.2 MB/s ,6589.7 MB/s
    12#movie.mkv         :1186646289 ->1186542565 (1.000),  82.9 MB/s ,6309.3 MB/s
    13#movie.mkv         :1186646289 ->1186543214 (1.000),  71.0 MB/s ,6465.4 MB/s
    14#movie.mkv         :1186646289 ->1186542413 (1.000),  72.4 MB/s ,6996.4 MB/s
    15#movie.mkv         :1186646289 ->1186539987 (1.000),  68.3 MB/s ,6447.2 MB/s
    16#movie.mkv         :1186646289 ->1186516228 (1.000),  21.8 MB/s ,6998.9 MB/s
    17#movie.mkv         :1186646289 ->1186496656 (1.000),  15.2 MB/s ,6577.2 MB/s    <17
    18#movie.mkv         :1186646289 ->1186419421 (1.000),  13.1 MB/s ,6836.3 MB/s    <18
    19#movie.mkv         :1186646289 ->1186045477 (1.001),  10.7 MB/s ,6174.1 MB/s    <19
    20#movie.mkv         :1186646289 ->1186164167 (1.000),  8.20 MB/s ,5676.2 MB/s    <20
    21#movie.mkv         :1186646289 ->1186256080 (1.000),  6.85 MB/s ,6320.4 MB/s    <21
    
    C:\m3>zstd144 -T4 -b1 -e21 movie.mkv
     1#movie.mkv         :1186646289 ->1186604884 (1.000),3043.7 MB/s ,7012.9 MB/s    < 1
     2#movie.mkv         :1186646289 ->1186602965 (1.000),3033.4 MB/s ,6987.5 MB/s
     3#movie.mkv         :1186646289 ->1186594049 (1.000),2239.0 MB/s ,6907.0 MB/s
     4#movie.mkv         :1186646289 ->1186592964 (1.000),1709.5 MB/s ,6958.6 MB/s
     5#movie.mkv         :1186646289 ->1186579688 (1.000), 197.8 MB/s ,6978.1 MB/s
     6#movie.mkv         :1186646289 ->1186570397 (1.000), 179.4 MB/s ,6995.2 MB/s
     7#movie.mkv         :1186646289 ->1186563987 (1.000), 178.1 MB/s ,6927.6 MB/s
     8#movie.mkv         :1186646289 ->1186563386 (1.000), 176.5 MB/s ,6923.8 MB/s
     9#movie.mkv         :1186646289 ->1186558263 (1.000), 125.1 MB/s ,6975.7 MB/s
    10#movie.mkv         :1186646289 ->1186539076 (1.000),  97.9 MB/s ,6898.2 MB/s    <10
    11#movie.mkv         :1186646289 ->1186537416 (1.000),  82.7 MB/s ,6204.1 MB/s    <11
    12#movie.mkv         :1186646289 ->1186537096 (1.000),  82.4 MB/s ,6416.6 MB/s    <12
    13#movie.mkv         :1186646289 ->1186537868 (1.000),  79.2 MB/s ,6965.1 MB/s    <13
    14#movie.mkv         :1186646289 ->1186537048 (1.000),  80.5 MB/s ,6888.5 MB/s    <14
    15#movie.mkv         :1186646289 ->1186536576 (1.000),  75.7 MB/s ,6897.0 MB/s    <15
    16#movie.mkv         :1186646289 ->1186521377 (1.000),  26.4 MB/s ,6931.9 MB/s    <16
    17#movie.mkv         :1186646289 ->1186493599 (1.000),  17.8 MB/s ,6567.4 MB/s    <17
    18#movie.mkv         :1186646289 ->1186416899 (1.000),  12.9 MB/s ,6747.5 MB/s    <18
    19#movie.mkv         :1186646289 ->1186049362 (1.001),  10.7 MB/s ,6108.6 MB/s    <19
    20#movie.mkv         :1186646289 ->1186167024 (1.000),  7.80 MB/s ,6292.4 MB/s
    21#movie.mkv         :1186646289 ->1186256011 (1.000),  6.42 MB/s ,6307.3 MB/s    <21
    
    CinebenchR20
    
    C:\m3>zstd144 -T2 -b1 -e21 CBR20.tar
     1#CBR20.tar         : 546450944 -> 242922546 (2.249), 761.6 MB/s , 992.8 MB/s
     2#CBR20.tar         : 546450944 -> 231262238 (2.363), 620.6 MB/s , 940.0 MB/s
     3#CBR20.tar         : 546450944 -> 220516212 (2.478), 415.2 MB/s , 931.5 MB/s
     4#CBR20.tar         : 546450944 -> 217928680 (2.507), 258.2 MB/s , 921.0 MB/s
     5#CBR20.tar         : 546450944 -> 213166583 (2.563), 111.8 MB/s , 885.2 MB/s
     6#CBR20.tar         : 546450944 -> 212191956 (2.575),  93.0 MB/s , 918.3 MB/s
     7#CBR20.tar         : 546450944 -> 208137062 (2.625),  76.4 MB/s , 938.9 MB/s
     8#CBR20.tar         : 546450944 -> 206313161 (2.649),  50.9 MB/s , 946.1 MB/s
     9#CBR20.tar         : 546450944 -> 206000095 (2.653),  54.2 MB/s , 960.5 MB/s
    10#CBR20.tar         : 546450944 -> 199165433 (2.744),  46.0 MB/s , 961.7 MB/s
    11#CBR20.tar         : 546450944 -> 198785534 (2.749),  40.9 MB/s , 915.1 MB/s
    12#CBR20.tar         : 546450944 -> 198737974 (2.750),  32.4 MB/s , 950.0 MB/s
    13#CBR20.tar         : 546450944 -> 198666797 (2.751),  24.6 MB/s , 920.2 MB/s
    14#CBR20.tar         : 546450944 -> 198156566 (2.758),  22.1 MB/s , 952.3 MB/s
    15#CBR20.tar         : 546450944 -> 197571506 (2.766),  17.9 MB/s , 953.4 MB/s
    16#CBR20.tar         : 546450944 -> 188269371 (2.902),  12.3 MB/s , 922.4 MB/s
    17#CBR20.tar         : 546450944 -> 182651131 (2.992),  9.71 MB/s , 861.6 MB/s
    18#CBR20.tar         : 546450944 -> 176640474 (3.094),  8.11 MB/s , 776.5 MB/s
    19#CBR20.tar         : 546450944 -> 176132097 (3.103),  6.73 MB/s , 753.8 MB/s
    20#CBR20.tar         : 546450944 -> 172622595 (3.166),  5.64 MB/s , 749.0 MB/s
    21#CBR20.tar         : 546450944 -> 171516404 (3.186),  5.16 MB/s , 749.7 MB/s
    
    C:\m3>zstd144 -T3 -b1 -e21 CBR20.tar
     1#CBR20.tar         : 546450944 -> 242922546 (2.249),1099.8 MB/s , 983.8 MB/s
     2#CBR20.tar         : 546450944 -> 231212913 (2.363), 884.1 MB/s , 936.3 MB/s    < 2
     3#CBR20.tar         : 546450944 -> 220516212 (2.478), 515.1 MB/s , 922.3 MB/s
     4#CBR20.tar         : 546450944 -> 217928680 (2.507), 248.2 MB/s , 906.3 MB/s
     5#CBR20.tar         : 546450944 -> 213166583 (2.563), 135.7 MB/s , 888.0 MB/s
     6#CBR20.tar         : 546450944 -> 212191956 (2.575), 105.1 MB/s , 883.3 MB/s
     7#CBR20.tar         : 546450944 -> 208137062 (2.625),  87.5 MB/s , 930.4 MB/s
     8#CBR20.tar         : 546450944 -> 206313161 (2.649),  77.6 MB/s , 951.1 MB/s
     9#CBR20.tar         : 546450944 -> 206000095 (2.653),  69.1 MB/s , 954.9 MB/s
    10#CBR20.tar         : 546450944 -> 199129929 (2.744),  59.0 MB/s , 951.9 MB/s    <10
    11#CBR20.tar         : 546450944 -> 198740601 (2.750),  56.4 MB/s , 949.0 MB/s    <11
    12#CBR20.tar         : 546450944 -> 198694780 (2.750),  42.6 MB/s , 958.0 MB/s    <12
    13#CBR20.tar         : 546450944 -> 198630978 (2.751),  36.1 MB/s , 941.9 MB/s    <13
    14#CBR20.tar         : 546450944 -> 198106730 (2.758),  31.5 MB/s , 953.8 MB/s    <14
    15#CBR20.tar         : 546450944 -> 197478987 (2.767),  26.5 MB/s , 964.4 MB/s    <15
    16#CBR20.tar         : 546450944 -> 188145058 (2.904),  18.5 MB/s , 919.5 MB/s    <16
    17#CBR20.tar         : 546450944 -> 182651131 (2.992),  13.4 MB/s , 838.2 MB/s
    18#CBR20.tar         : 546450944 -> 176640474 (3.094),  11.2 MB/s , 754.9 MB/s
    19#CBR20.tar         : 546450944 -> 176132097 (3.103),  9.03 MB/s , 715.2 MB/s
    20#CBR20.tar         : 546450944 -> 172759205 (3.163),  7.69 MB/s , 751.6 MB/s    <20
    21#CBR20.tar         : 546450944 -> 171538529 (3.186),  6.79 MB/s , 744.6 MB/s    <21
    
    C:\m3>zstd144 -T4 -b1 -e21 CBR20.tar
     1#CBR20.tar         : 546450944 -> 242911513 (2.250),1418.9 MB/s , 977.5 MB/s    < 1
     2#CBR20.tar         : 546450944 -> 231096406 (2.365),1099.5 MB/s , 938.0 MB/s    < 2
     3#CBR20.tar         : 546450944 -> 220343277 (2.480), 533.6 MB/s , 901.3 MB/s    < 3
     4#CBR20.tar         : 546450944 -> 217998105 (2.507), 220.7 MB/s , 905.9 MB/s    < 4
     5#CBR20.tar         : 546450944 -> 213189073 (2.563), 152.7 MB/s , 899.5 MB/s    < 5
     6#CBR20.tar         : 546450944 -> 212217325 (2.575), 120.2 MB/s , 910.1 MB/s    < 6
     7#CBR20.tar         : 546450944 -> 208162494 (2.625), 100.6 MB/s , 932.3 MB/s    < 7
     8#CBR20.tar         : 546450944 -> 206283262 (2.649),  87.8 MB/s , 950.1 MB/s    < 8
     9#CBR20.tar         : 546450944 -> 205975084 (2.653),  80.1 MB/s , 955.4 MB/s    < 9
    10#CBR20.tar         : 546450944 -> 199159623 (2.744),  68.0 MB/s , 950.3 MB/s    <10
    11#CBR20.tar         : 546450944 -> 198776880 (2.749),  63.5 MB/s , 953.1 MB/s    <11
    12#CBR20.tar         : 546450944 -> 198729782 (2.750),  46.1 MB/s , 953.4 MB/s    <12
    13#CBR20.tar         : 546450944 -> 198657903 (2.751),  40.9 MB/s , 961.3 MB/s    <13
    14#CBR20.tar         : 546450944 -> 198147016 (2.758),  37.2 MB/s , 963.4 MB/s    <14
    15#CBR20.tar         : 546450944 -> 197559001 (2.766),  29.8 MB/s , 970.7 MB/s    <15
    16#CBR20.tar         : 546450944 -> 188375448 (2.901),  22.4 MB/s , 928.6 MB/s    <16
    17#CBR20.tar         : 546450944 -> 182589644 (2.993),  16.4 MB/s , 890.2 MB/s    <17
    18#CBR20.tar         : 546450944 -> 176558517 (3.095),  13.9 MB/s , 776.2 MB/s    <18
    19#CBR20.tar         : 546450944 -> 176085832 (3.103),  11.2 MB/s , 764.1 MB/s    <19
    20#CBR20.tar         : 546450944 -> 172633923 (3.165),  9.80 MB/s , 753.7 MB/s    <20
    21#CBR20.tar         : 546450944 -> 171538529 (3.186),  6.86 MB/s , 748.4 MB/s    <21
    Last edited by Piglet; 17th May 2020 at 14:55.

  20. #404
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    768
    Thanks
    236
    Thanked 245 Times in 150 Posts
    Introduce ZSTD compression to ZFS: https://github.com/openzfs/zfs/pull/10278
    https://news.ycombinator.com/item?id=23210491
    https://www.bsdcan.org/2018/schedule/events/947.en.html
    https://www.phoronix.com/scan.php?pa...For-ZFS-Review

    ps. 6 month old 1.4.4 ... but 1.4.5 is coming: https://github.com/facebook/zstd/com...1400a62d15aca6
    Starting with
    "perf: Improved decompression speed: x64 : +10% (clang) / +5% (gcc); ARM : from +15% to +50%, depending on SoC, by @terrelln"

  21. #405
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    883
    Thanks
    478
    Thanked 277 Times in 117 Posts
    Thanks for feedback @Piglet.

    This is a very subtle issue.
    The `zstd` CLI guarantees reproducibility of the compressed result whatever the nb of threads.
    This is true because it uses the "streaming" API, which guarantees this property.

    If you try to reproduce this outcome by actually compressing the file, and counting the bytes produced,
    you'll notice that the total produced is rigorously identical, whatever the nb of threads.

    However, the benchmark module uses the simpler one-shot ingestion API
    (entire input present in memory, entire output produced immediately into another buffer),
    because it measures mem-to-mem compression and decompression speeds.
    The one-shot ingestion API is more convenient, and more efficient for small inputs,
    but it is unwieldy for large inputs, due to higher latency and higher memory requirements.

    The one-shot ingestion API _can_ result in a slightly different outcome with different amount of threads, as it _may_ split the jobs differently.
    Note though that using the same nb of threads always leads to the same exact outcome,
    so even this mode is still "reproducible", it's just that the nb of threads is now part of the parameters to capture in order to achieve reproducibility.

    The one-shot ingestion API is never used when compressing with the `zstd` CLI,
    so the advertised CLI reproducibility guarantee independent of selected nb of threads still holds in this context.

  22. Thanks:

    Piglet (18th May 2020)

  23. #406
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    280
    Thanks
    109
    Thanked 51 Times in 35 Posts
    Quote Originally Posted by Jarek View Post
    Introduce ZSTD compression to ZFS: https://github.com/openzfs/zfs/pull/10278
    https://news.ycombinator.com/item?id=23210491
    https://www.bsdcan.org/2018/schedule/events/947.en.html
    https://www.phoronix.com/scan.php?pa...For-ZFS-Review

    ps. 6 month old 1.4.4 ... but 1.4.5 is coming: https://github.com/facebook/zstd/com...1400a62d15aca6
    Starting with
    "perf: Improved decompression speed: x64 : +10% (clang) / +5% (gcc); ARM : from +15% to +50%, depending on SoC, by @terrelln"
    Jarek, have you compiled it with Visual Studio? I wonder why the benchmarks only report clang and gcc results. Anyone try building Zstd in Visual Studio? What about compiler optimizations?

  24. #407
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    768
    Thanks
    236
    Thanked 245 Times in 150 Posts
    I was just quoting, but I see benchmarks are updated e.g. in https://facebook.github.io/zstd/

    zstd 1.4.5 -1 2.884 500 MB/s 1660 MB/s
    zlib 1.2.11 -1 2.743 90 MB/s 400 MB/s
    brotli 1.0.7 -0 2.703 400 MB/s 450 MB/s
    zstd 1.4.5 --fast=1 2.434 570 MB/s 2200 MB/s
    zstd 1.4.5 --fast=3 2.312 640 MB/s 2300 MB/s
    quicklz 1.5.0 -1 2.238 560 MB/s 710 MB/s
    zstd 1.4.5 --fast=5 2.178 700 MB/s 2420 MB/s
    lzo1x 2.10 -1 2.106 690 MB/s 820 MB/s
    lz4 1.9.2 2.101 740 MB/s 4530 MB/s
    lzf 3.6 -1 2.077 410 MB/s 860 MB/s
    snappy 1.1.8 2.073 560 MB/s 1790 MB/s

  25. #408
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    812
    Thanks
    234
    Thanked 292 Times in 174 Posts
    No one I know uses brotli at -0 or zlib at -1. These settings are too fast for practical compression. The fastest brotli settings in use that I know of is -2, for one use machine to machine transport. If a human is waiting for the data then a -5 to -7 can be justified for one use only use case, too.

  26. #409
    Member
    Join Date
    Apr 2015
    Location
    Greece
    Posts
    90
    Thanks
    34
    Thanked 26 Times in 17 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    No one I know uses brotli at -0 or zlib at -1. These settings are too fast for practical compression. The fastest brotli settings in use that I know of is -2, for one use machine to machine transport. If a human is waiting for the data then a -5 to -7 can be justified for one use only use case, too.
    Transparent compression in filesystems is a big thing and apparently they use fast modes. You may think 400MB/s is a lot for spinning disk. But it is slow for SSD.

    And there is RAID. 10s of disks per node and CPU can start to be a bottleneck even for HDD.

  27. #410
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    611
    Thanks
    246
    Thanked 240 Times in 119 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    These settings are too fast for practical compression.
    One interesting use case I've been co-working on lately is this screen sharing software. It allows to use another PC (e.g. an old laptop) as desktop extension (so it basically gives the same functionality as an attached monitor) by transferring the output of a virtual desktop to the other computer over a direct 1 GB/s Ethernet cable connection. Of course this can be done using lossy compression and ignoring the artifacts, but it's interesting that lz4 (that is also listed in the table above) gives good results for this case on compression ratio (how much has to be transfered, limited by the ethernet connection), decompression speed (so latency will be low and no framedrops will occur) and CPU usage.
    The requirements to the (de)compression algorithm for this case are very atypical, but they exist (and they can be scaled - FullHD - or even 4K? 30Hz or 60Hz? Empty desktop or web browser playing a video? Can we do this with 100 MBit/s too while keeping it lossless?).

    Apart from that, the table is from a benchmark, so the main question shouldn't be if people use these settings, they're useful to compare different algorithms against each other.
    http://schnaader.info
    Damn kids. They're all alike.

  28. #411
    Member
    Join Date
    Nov 2014
    Location
    California
    Posts
    144
    Thanks
    49
    Thanked 40 Times in 29 Posts
    These settings are too fast for practical compression.
    I am sure that the guys integrating zstd in ZFS or databases beg to differ. And zstd 1.3.4 introduced faster modes.
    Brotli performs poorly at level 0, so this level should be either removed (if unused) or fixed.

  29. #412
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    280
    Thanks
    109
    Thanked 51 Times in 35 Posts
    I've always found it strange that the Zstd benchmarks use such weird settings. -1 and -0 are not typical modes for Zstd and brotli. It's not clear whether Zstd is meaningfully faster or more compressive than brotli given those settings, especially given that Zstd has 22 levels and brotli has 11, not counting any subzero levels.

    ​We probably need a graph that presents all levels, or at least a table that gives three or four levels each.

  30. #413
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    768
    Thanks
    236
    Thanked 245 Times in 150 Posts
    https://github.com/openzfs/zfs/pull/10278 has these nice graphs presenting also fast levels:


  31. #414
    Member
    Join Date
    Apr 2015
    Location
    Greece
    Posts
    90
    Thanks
    34
    Thanked 26 Times in 17 Posts
    Also the misleading thing I think is that they test on a 5GHz CPU.

    This overestimates the performance of all compressors.

  32. #415
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    937
    Thanks
    95
    Thanked 363 Times in 252 Posts
    Quote Originally Posted by algorithm View Post
    that they test on a 5GHz CPU.
    Code:
    i9-10900T,  1.9GHz base, 4.6GHz turbo boost, 4.6GHz max turbo, 35 Watt, micro desktop
    i9-10980HK, 2.4GHz base, 5.1GHz turbo boost, 5.3GHz max turbo, 45 Watt, mobile
    i9-10900,   2.8GHz base, 5.1GHz turbo boost, 5.2GHz max turbo, 65 Watt, mini desktop
    i9-10900K,  3.7GHz base, 5.2GHz turbo boost, 5.3GHz max turbo, 125 Watt, desktop
    
    i7-10700T,  2.0GHz base, 4.5GHz turbo boost, 4.5GHz max turbo, 35 Watt, micro desktop
    i7-10750H,  2.6GHz base, 4.8GHz turbo boost, 5.0GHz max turbo, 45 Watt, mobile
    i7-10700,   2.9GHz base, 4.8GHz turbo boost, 4.8GHz max turbo, 65 Watt, mini desktop
    i7-10700K,  3.8GHz base, 5.1GHz turbo boost, 5.1GHz max turbo, 125 Watt, desktop
    
    i5-10600,   3.3GHz base,                   , 4.8GHz max turbo, 65 Watt, mini desktop
    i5-10600K,  4.1GHz base,                   , 4.8GHz max turbo, 125 Watt, desktop

  33. #416
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    812
    Thanks
    234
    Thanked 292 Times in 174 Posts
    I would use a simple non-context modeling compressor for usual compressed filesystems, not brotli.

  34. #417
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    812
    Thanks
    234
    Thanked 292 Times in 174 Posts
    100 mbit/s is only 10 MB/s. There you can safely use brotli 7 and get the compression done by a single core.

  35. #418
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    883
    Thanks
    478
    Thanked 277 Times in 117 Posts
    @Kirr made an impressive data compression comparison website

  36. Thanks (2):

    Kirr (22nd May 2020),schnaader (22nd May 2020)

  37. #419
    Member
    Join Date
    May 2019
    Location
    Japan
    Posts
    25
    Thanks
    4
    Thanked 8 Times in 4 Posts
    Quote Originally Posted by Cyan View Post
    @Kirr made an impressive data compression comparison website
    Thanks Yann! Congrats with releasing 1.4.5! I am already testing it, should have results in a few days.

  38. Thanks:

    Cyan (23rd May 2020)

  39. #420
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    280
    Thanks
    109
    Thanked 51 Times in 35 Posts
    Can someone explain the new "patch size" feature in Zstd 1.4.5? I don't understand it. Why are the percentages less than 1% in the graph? What do these values represent? (Graph screenshot below – FYI, Yann, you can save 14% on the png by losslessly optimizing it...)

    Click image for larger version. 

Name:	patch-size-bsdiff-vs-zstd-19.png 
Views:	16 
Size:	14.9 KB 
ID:	7622

Page 14 of 15 FirstFirst ... 412131415 LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •