Page 7 of 8 FirstFirst ... 5678 LastLast
Results 181 to 210 of 224

Thread: Brotli

  1. #181
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    878
    Thanks
    80
    Thanked 315 Times in 219 Posts
    Introducing Brotli compression in Microsoft Edge:
    https://blogs.windows.com/msedgedev/...i-compression/

  2. #182
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    239
    Thanks
    95
    Thanked 47 Times in 31 Posts
    Quote Originally Posted by Sportman View Post
    Introducing Brotli compression in Microsoft Edge:
    https://blogs.windows.com/msedgedev/...i-compression/
    By the way, this was finally released to the public on April 11 as part of the Windows 10 Creators Update. Until now, it was only in the beta versions of Edge that Windows Insiders use.

    Brotli is pretty disappointing, and instead of wasting time with it I wish these companies would collaborate to build something that would deliver much larger compression gains, and much lower CPU processing demands during the compression phase. Brotli was developed in isolation, without any peer review, and it's clear that we could do better if we started over.

  3. #183
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    239
    Thanks
    95
    Thanked 47 Times in 31 Posts

    Brotli 0.6.0 released

    Brotli has had a new release, the first in eight months: 0.6.0. It looks like it will deliver better compression ratios for some files, and at some settings.

    The brotli project doesn't release any executables. Most of the files are Python wheels. I'll try building in Visual Studio, but hopefully Load will do it sooner. Load, could you try it with CPU optimizations, like march=westmere? And I wonder if the linker optimizations will help (-flto in gcc).

  4. #184
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    239
    Thanks
    95
    Thanked 47 Times in 31 Posts

    Updated brotli build

    Hi all – I figured out how to compile it with cmake. I've attached the Win64 executable. Actually, I zipped the whole Release folder with the DLLs because I'm not sure if they're needed to run bro.exe.

    This was compiled in Visual Studio 2017 on Windows 10, with default settings using the CMakeLists.txt that came in the brotli master repo. Note that this build might be different/newer than 0.6.0 because the 0.6.0 release did not include a CMakeLists.txt file. So I downloaded a zip of the master instead. (0.6.0 was released on April 9, 2017)

    I haven't tried any compiler optimizations yet, but eventually I might see if setting the CPU floor at Haswell impacts anything for the compiler and the resultant bro.exe performance. I'm also going to try the Intel compiler once they fix integration with Visual Studio 2017.
    Attached Files Attached Files

  5. #185
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    204
    Thanks
    170
    Thanked 109 Times in 65 Posts
    a static gcc 6.3 compile

    untested, no flags added. i guess the VS compile will be faster.
    Attached Files Attached Files

  6. #186
    Member
    Join Date
    Dec 2016
    Location
    Norway
    Posts
    18
    Thanks
    15
    Thanked 10 Times in 4 Posts
    Did some testing with SolidComp compiles of brotli 0.6.0. Compared with lzma2 ultra in 7zip GUI. Brotli is waaay slower (--quality 11) and does not produce smaller files.
    Used bro.exe --quality 11 --input "path\file" --output "path\file.br"
    - Python.tar is a folder with python flask websites
    - test.wav is a 44.1khz 16bit wav
    - n64-rom is a n64 rom file
    - pnbwiki.xml is the database dump of pnbwiki (https://dumps.wikimedia.org/pnbwiki/...stream.xml.bz2)
    Click image for larger version. 

Name:	brotli.PNG 
Views:	120 
Size:	12.1 KB 
ID:	4892

  7. #187
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    486
    Thanks
    168
    Thanked 166 Times in 114 Posts
    IIRC brotli's design was to have fast decompression, as it was intended to be used in serving up static web pages. Lzma is still reasonable decompression speeds, but not up to, say, zstd levels. I don't know how Brotli compares at decompression times.

  8. Thanks:

    load (27th April 2017)

  9. #188
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    204
    Thanks
    170
    Thanked 109 Times in 65 Posts
    some more compiles, gcc 6.3, only x64, but this time with flags

    -O3 -march=westmere
    -O3 -march=haswell
    -O3 -march=skylake
    -O3 -march=ivybridge

    might be faster.

    /edit:
    IIRC brotli's design was to have fast decompression, as it was intended to be used in serving up static web pages.
    i remember the same, also it's not meant for big files. so no competitor to 7z or rar or whatever. maybe the reason why they don't offer binaries.
    Attached Files Attached Files
    Last edited by load; 27th April 2017 at 17:20.

  10. #189
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    749
    Thanks
    215
    Thanked 282 Times in 164 Posts
    Quote Originally Posted by olavrb View Post
    Did some testing with SolidComp compiles of brotli 0.6.0. Compared with lzma2 ultra in 7zip GUI.
    If you are interested in unlimited window size use case, try out with the "Large Window Brotli". For example, enwik9 compressed about 10 % more densely with it.

  11. #190
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    239
    Thanks
    95
    Thanked 47 Times in 31 Posts
    Quote Originally Posted by olavrb View Post
    Did some testing with SolidComp compiles of brotli 0.6.0. Compared with lzma2 ultra in 7zip GUI. Brotli is waaay slower (--quality 11) and does not produce smaller files.
    Used bro.exe --quality 11 --input "path\file" --output "path\file.br"
    - Python.tar is a folder with python flask websites
    - test.wav is a 44.1khz 16bit wav
    - n64-rom is a n64 rom file
    - pnbwiki.xml is the database dump of pnbwiki (https://dumps.wikimedia.org/pnbwiki/...stream.xml.bz2)
    Click image for larger version. 

Name:	brotli.PNG 
Views:	120 
Size:	12.1 KB 
ID:	4892

    Brotli has always been super slow at -11. It's sort of expected that -11 will be super slow – it's practically a different codec at -11 (and -10) then at all the lower values. Brotli -11 usually beats Zstd -22 in compression ratio, but Zstd will be several times faster at those settings.

    I'm surprised brotli didn't do well with the XML file since it has a built-in text dictionary. The dictionary is a terrible mess, with too-short strings, but it does seem to help most of the time with web content, HTML, etc. I'm not surprised it doesn't do well on the wav, rom, etc. since the dictionary is only for text. Actually, it would be pretty interesting to build a codec that had a multifaceted, multimedia dictionary. Has this been done already? It would have long useful HTML strings, and common binary sequences for multimedia. I wonder if there are common byte sequences – maybe just in header and other structural fields.

  12. #191
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    239
    Thanks
    95
    Thanked 47 Times in 31 Posts

    Optimized Visual Studio and Intel Compiler builds

    I been playing with Visual Studio 2017 and the new Intel Parallel Studio 2018 Beta. I've attached brotli builds from each compiler. Notes:

    For the Visual Studio build I've made several changes since the build I posted three days ago:

    1. I enabled link-time optimization, aka Whole Program Optimization, for all the components (encoder, decoder, common, etc.)
    2. I enabled Control Flow Guard, which makes this build significantly safer than a vanilla build. Compression software is dangerous because it often processes arbitrary, untrusted input. Google's Project Zero has done a great job of exposing how terrible a lot of antivirus software is, and one of their big findings was how Symantec's endpoint software exposed users to easy breaches because of vulnerabilities in its UPX unpacker, among other things.
    3. I enabled a CPU floor in the code generation options, specifically AVX. This means the build is for Sandy Bridge and up – it may or may not work for pre-Sandy Bridge machines. (I suspect it will work for older machines, because I don't think Visual Studio actually performed any automatic vectorization – i.e. I don't think it generated any AVX or other new-to-Sandy-Bridge instructions).
    4. I chose to prioritize fast vs. small code.
    5. I chose fast, non-strict floating point.

    I don't think 3 and 5 will make much of a difference. Unless brotli was written in such a way to ease auto-vectorization, I don't think today's compilers will auto-vectorize it much. And I don't think brotli has much floating point action (most compression codecs don't, right?)

    The linker optimizations might make a difference. Control Flow Guard will probably cause a single-digit percentage slowdown due to its overhead. I have no idea whether choosing speed over size will matter for brotli in this case – it will be interesting to find out.

    On the Intel compiler, I enabled the linker optimizations (multi-file), and a few other things, and set the CPU floor to Ivy Bridge (which will be AVX + a few other instructions, but before AVX2). In the Intel 7zip file I've attached, I included the screenshots for all the Intel optimization windows/pop-ups, so you could see the sorts of things that are in play. One window is all about Intel Performance Libraries, and as you can see in that screenshot, I didn't enable any of those libraries. I have no idea if any of them make sense for brotli. I'd be very interested in what Jyrki or others familiar with brotli's code think about the potential of those Intel library options. (They also offer "Intel-optimized headers" – I don't remember if I enabled that.)

    Note that Control Flow Guard was not an option when using the Intel compiler – it's a Visual Studio compiler feature. So the Intel builds are less safe in that respect. All my builds have enabled (64-bit) ASLR and Data Execution Prevention, which are defaults now in Visual Studio.

  13. #192
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,532
    Thanks
    755
    Thanked 674 Times in 365 Posts
    1. afaik Control Flow Guard require MPX cpu extension, i.e. skylake+
    2. AVX build is hardly fas
    ter than SSE2, but it will fail on older cpus since compiler may still generate a few AVX commands here and there, f.e. when performing FP calculations

  14. #193
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    749
    Thanks
    215
    Thanked 282 Times in 164 Posts
    We have spent most time in density optimizations of level 11, and speed of decoding on gcc/Linux, both Arm and Intel. We spent some time on the speed of encoding at levels 0 and 1. Recently, we optimized long data (1 MB+) by ~10% in density for levels 4–9. We are likely going to improve on density of levels 4–9 by 2–3 % more during the next year, by more advanced context modeling (possibly less for short data). Also, we plan optimizations targeting the encoding speed gap between levels 9 and 10.

    We haven't done performance testing and optimizations on Windows, and it is very helpful when someone else is analyzing it on Windows.

  15. #194
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    749
    Thanks
    215
    Thanked 282 Times in 164 Posts
    A better answer to the question: Probably encoding speed of levels 10 and 11 is easiest to influence by smarter use of data parallel instructions.

  16. #195
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    239
    Thanks
    95
    Thanked 47 Times in 31 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    1. afaik Control Flow Guard require MPX cpu extension, i.e. skylake+
    2. AVX build is hardly fas
    ter than SSE2, but it will fail on older cpus since compiler may still generate a few AVX commands here and there, f.e. when performing FP calculations
    Control Flow Guard doesn't require MPX. I don't think it uses MPX if it's​ available. The latest Intel instructions are too murky and poorly documented to use in actual software. I mean things like MPX, SGX, TSX, etc. They don't even include TSX on some Skylake and Kaby Lake CPUs, and it's hard to know which models have it. It's a mess.

  17. #196
    Member
    Join Date
    May 2012
    Location
    Germany
    Posts
    39
    Thanks
    25
    Thanked 103 Times in 24 Posts
    I have just released a new version of the zstdmt library.

    This version includes brotli v0.6.0 as a testing tool with the full gzip/bzip2 like interface. Multithreading is done via encapsulating brotli chunks with skippable frames.


    best regards, Tino

  18. Thanks:

    Jyrki Alakuijala (22nd March 2018)

  19. #197
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    474
    Thanked 175 Times in 85 Posts
    Can anybody please post a fresh build of Brotli 1.0.3 here?
    I cannot compile it for Win 64-bit..

  20. #198
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    204
    Thanks
    170
    Thanked 109 Times in 65 Posts
    brotli 1.03 x64 windows exe

    see below
    Last edited by load; 22nd March 2018 at 18:44.

  21. Thanks (3):

    hunman (22nd March 2018),Sportman (22nd March 2018),Stephan Busch (22nd March 2018)

  22. #199
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    474
    Thanked 175 Times in 85 Posts
    that exe triggered a virus alert and Windows Defender deleted it

  23. #200
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    204
    Thanks
    170
    Thanked 109 Times in 65 Posts
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	- virusscan_jotti_org_de-DE_filescanjob_y7kq8e3n7u.png 
Views:	99 
Size:	115.4 KB 
ID:	5850   Click image for larger version. 

Name:	- www_virustotal_com_de_file_423060862c7b8fcae9c7ea2251ba87322e24c18d0174c7d8cc03ede8e51b6b1d_an.png 
Views:	55 
Size:	131.9 KB 
ID:	5851  

  24. #201
    Member
    Join Date
    May 2017
    Location
    Hungary
    Posts
    7
    Thanks
    43
    Thanked 3 Times in 3 Posts

  25. Thanks:

    Stephan Busch (22nd March 2018)

  26. #202
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    878
    Thanks
    80
    Thanked 315 Times in 219 Posts
    Code:
    100,000,000 bytes - enwik8 HTML:
    
          C Size  ratio%     C MB/s     D MB/s   Name
        25380948    25.4       1.93     789.10   zstd 22                          
        25581278    25.6       2.07     797.84   zstd 21                          
        26028942    26.0       2.32     826.28   zstd 20                          
        27044781    27.0       0.67     544.84   brotli 11                        
        27549634    27.5       1.37     469.23   brotli 10                        
        27658726    27.7       3.59    1088.01   zstd 19                          
        28241804    28.2       4.63    1093.58   zstd 18                          
        28587027    28.6       5.15    1094.49   zstd 17                          
        29092452    29.1       7.19    1198.70   zstd 16                          
        29468810    29.5       8.48     562.08   brotli 9                         
        29774467    29.8       7.05    1134.91   zstd 15                          
        29806381    29.8      11.66     559.15   brotli 8                         
        30306203    30.3      16.62     549.14   brotli 7                         
        30336988    30.3      10.78    1116.89   zstd 14                          
        30444985    30.4      12.34    1112.50   zstd 13                          
        30991445    31.0      25.73     537.09   brotli 6                         
        31056443    31.1      17.32    1196.99   zstd 12                          
        31230227    31.2      24.85    1101.33   zstd 11                          
        31746267    31.7      32.66    1195.73   zstd 10                          
        31776975    31.8      34.97     518.82   brotli 5                         
        31992723    32.0      49.51    1198.77   zstd 9                           
        32257694    32.3      55.45    1097.16   zstd 8                           
        32737388    32.7      68.77    1055.72   zstd 7                           
        33304947    33.3      93.05    1036.07   zstd 6                           
        33515149    33.5      70.06     555.80   brotli 4                         
        34736348    34.7     123.32     990.54   zstd 5                           
        35205471    35.2     190.80    1027.37   zstd 4                           
        35687382    35.7     204.05    1048.35   zstd 3                           
        36475796    36.5      23.61     360.43   zlib 9                           
        36476038    36.5      23.67     360.44   zlib 8                           
        36498577    36.5      26.22     360.29   zlib 7                           
        36548925    36.5      29.31     359.81   zlib 6                           
        36685028    36.7     101.88     485.77   brotli 3                         
        36879096    36.9      40.76     355.71   zlib 5                           
        36936891    36.9     119.27     472.82   brotli 2                         
        37761110    37.8     254.71    1112.78   zstd 2                           
        37989516    38.0      60.38     358.60   zlib 4                           
        39123260    39.1     204.26     440.37   brotli 1                         
        39542367    39.5      61.71     371.88   zlib 3                           
        40746375    40.7      78.97     362.44   zlib 2                           
        40859471    40.9     371.56    1239.39   zstd 1                           
        41694594    41.7     264.04     406.38   brotli 0                         
        42298778    42.3      87.55     352.05   zlib 1 
                              
    
    27,617 byte - Binance homepage HTML:
    
          C Size  ratio%     C MB/s     D MB/s   Name
            6159    22.3       1.14     476.16   brotli 11                        
            6322    22.9       2.69     438.37   brotli 10                        
            6774    24.5       6.23     511.43   brotli 9                         
            6778    24.5       8.58     511.43   brotli 8                         
            6782    24.6      12.97     502.13   brotli 7                         
            6996    25.3      30.58     575.35   brotli 6                         
            7029    25.5      40.49     563.61   brotli 5                         
            7399    26.8      10.32    1380.85   zstd 22                          
            7399    26.8      10.33    1380.85   zstd 20                          
            7399    26.8      10.32    1380.85   zstd 21                          
            7408    26.8      14.89    1380.85   zstd 18                          
            7408    26.8      14.89    1380.85   zstd 19                          
            7413    26.8      15.45    1380.85   zstd 17                          
            7427    26.9      16.76    1380.85   zstd 16                          
            7473    27.1      18.65    1315.10   zstd 15                          
            7618    27.6      31.07    1453.53   zstd 13                          
            7630    27.6      21.64    1315.10   zstd 14                          
            7635    27.6      66.71    1453.53   zstd 11                          
            7636    27.6      73.45    1453.53   zstd 10                          
            7636    27.6      83.18    1453.53   zstd 9                           
            7637    27.7      61.24    1453.53   zstd 12                          
            7657    27.7      49.49     445.44   zlib 9                           
            7657    27.7      51.05     445.44   zlib 8                           
            7662    27.7      96.90    1453.53   zstd 8                           
            7662    27.7      95.89    1453.53   zstd 7                           
            7666    27.8      61.37     445.44   zlib 7                           
            7669    27.8      68.53     445.44   zlib 6                           
            7693    27.9      80.75     531.10   brotli 4                         
            7709    27.9     136.72    1380.85   zstd 6                           
            7724    28.0      83.44     438.37   zlib 5                           
            7803    28.3     170.48    1380.85   zstd 5                           
            7870    28.5     103.82     438.37   zlib 4                           
            7907    28.6     195.87    1380.85   zstd 4                           
            8167    29.6     104.61     575.35   brotli 3                         
            8261    29.9     121.13     552.34   brotli 2                         
            8305    30.1     306.86    1453.53   zstd 3                           
            8400    30.4     122.74     424.88   zlib 3                           
            8542    30.9     400.25    1380.85   zstd 2                           
            8543    30.9     138.78     424.88   zlib 2                           
            8662    31.4     368.23    1453.53   zstd 1                           
            8750    31.7     150.91     412.19   zlib 1                           
            8936    32.4     303.48     531.10   brotli 1                         
            9125    33.0     317.44     493.16   brotli 0 
                            
    
    844,524 bytes - Techcrunch homepage HTML:
    
          C Size  ratio%     C MB/s     D MB/s   Name
          101652    12.0       1.01     712.68   brotli 11                        
          103979    12.3       2.82     668.67   brotli 10                        
          113217    13.4       6.03    2246.07   zstd 22                          
          113295    13.4       6.84    2246.07   zstd 21                          
          113519    13.4       8.68    2240.12   zstd 20                          
          113588    13.4      11.79    2240.12   zstd 19                          
          114664    13.6      12.59    2222.43   zstd 18                          
          115146    13.6      13.67    2288.68   zstd 17                          
          115214    13.6      29.41     880.63   brotli 9                         
          115734    13.7      40.11     880.63   brotli 8                         
          115964    13.7      15.07    2352.43   zstd 16                          
          116167    13.8      51.30     880.63   brotli 7                         
          117438    13.9      68.28     871.54   brotli 6                         
          117666    13.9      18.38    2406.05   zstd 15                          
          117739    13.9      21.45    2392.42   zstd 14                          
          118020    14.0      26.79    2385.66   zstd 13                          
          118574    14.0      64.38    2392.42   zstd 12                          
          119079    14.1      80.72    2378.94   zstd 11                          
          119113    14.1      86.81    2378.94   zstd 10                          
          119285    14.1      80.00     855.65   brotli 5                         
          120016    14.2     105.91    2345.90   zstd 9                           
          120255    14.2     113.99    2339.40   zstd 8                           
          121397    14.4     141.96    2294.90   zstd 7                           
          123792    14.7     171.03    2240.12   zstd 6                           
          127644    15.1     227.14    2165.45   zstd 5                           
          128454    15.2     147.57     841.16   brotli 4                         
          130540    15.5     375.51    2210.80   zstd 4                           
          130928    15.5     406.22    2210.80   zstd 3                           
          134817    16.0     205.43     765.66   brotli 3                         
          136783    16.2     560.03    2222.43   zstd 2                           
          139376    16.5     238.50     740.81   brotli 2                         
          143555    17.0     721.82    2282.50   zstd 1                           
          150709    17.8      59.29     624.65   zlib 9                           
          150713    17.8      59.55     625.11   zlib 8                           
          151008    17.9      63.58     624.65   zlib 7                           
          151158    17.9     460.73     723.67   brotli 1                         
          151543    17.9      68.52     623.26   zlib 6                           
          155508    18.4      83.78     609.76   zlib 5                           
          161415    19.1     105.98     600.23   zlib 4                           
          167293    19.8     551.26     701.43   brotli 0                         
          175280    20.8     135.34     592.65   zlib 3                           
          180142    21.3     147.39     578.84   zlib 2                           
          189064    22.4     154.84     558.92   zlib 1                           
    
    
    216,468 bytes - Soho homepage HTML:
    
          C Size  ratio%     C MB/s     D MB/s   Name
           30452    14.1       0.98     721.56   brotli 11                        
           30882    14.3       2.75     680.72   brotli 10                        
           32354    14.9      19.66     754.24   brotli 9                         
           32475    15.0      30.83     756.88   brotli 8                         
           32595    15.1      43.50     751.63   brotli 7                         
           33642    15.5      66.75     756.88   brotli 6                         
           33953    15.7      74.98     743.88   brotli 5                         
           35873    16.6       6.19    1898.84   zstd 21                          
           35873    16.6       6.20    1898.84   zstd 22                          
           35873    16.6       6.19    1898.84   zstd 20                          
           35921    16.6      11.65    1915.65   zstd 19                          
           35946    16.6      12.29    1915.65   zstd 18                          
           36030    16.6      13.58    1915.65   zstd 17                          
           36457    16.8      14.95    1882.33   zstd 16                          
           37276    17.2      24.48    2143.25   zstd 13                          
           37429    17.3      40.11    2143.25   zstd 12                          
           37469    17.3      70.30    2143.25   zstd 11                          
           37515    17.3      19.56    2101.63   zstd 14                          
           37541    17.3      76.98    2143.25   zstd 10                          
           37672    17.4      93.47    2122.24   zstd 9                           
           37727    17.4      17.95    1803.90   zstd 15                          
           37851    17.5     109.22    2101.63   zstd 8                           
           37922    17.5     130.25    2081.42   zstd 7                           
           38796    17.9     162.03    2186.55   zstd 6                           
           38955    18.0     139.84     883.54   brotli 4                         
           39034    18.0     194.84    2143.25   zstd 5                           
           39608    18.3      48.38     670.18   zlib 9                           
           39638    18.3      64.85     668.11   zlib 8                           
           39775    18.4      87.11     666.06   zlib 7                           
           39950    18.5      96.38     666.06   zlib 6                           
           40575    18.7     205.57     883.54   brotli 3                         
           40774    18.8     233.01    1985.94   zstd 4                           
           40819    18.9     111.07     652.01   zlib 5                           
           41382    19.1     438.19    1985.94   zstd 3                           
           41691    19.3     250.54     845.58   brotli 2                         
           41768    19.3     524.14    1985.94   zstd 2                           
           42164    19.5     124.12     632.95   zlib 4                           
           42923    19.8     627.44    1985.94   zstd 1                           
           43176    19.9     167.80     620.25   zlib 3                           
           43768    20.2     399.39     848.89   brotli 1                         
           44130    20.4     177.58     606.35   zlib 2                           
           45259    20.9     180.54     594.69   zlib 1                           
           48055    22.2     661.98     773.10   brotli 0                         
    
    
    689,827 bytes - 163 homepage HTML:
    
          C Size  ratio%     C MB/s     D MB/s   Name
           74941    10.9       0.89     740.95   brotli 11                        
           76479    11.1       2.58     712.63   brotli 10                        
           81408    11.8      32.52     749.81   brotli 9                         
           81973    11.9      47.60     752.26   brotli 8                         
           82333    11.9      61.04     744.15   brotli 7                         
           83027    12.0       5.92    2454.90   zstd 22                          
           83163    12.1       7.07    2446.20   zstd 21                          
           83444    12.1       9.10    2437.55   zstd 20                          
           83554    12.1      13.75    2446.20   zstd 19                          
           84270    12.2      15.55    2420.45   zstd 18                          
           84900    12.3      18.32    2481.39   zstd 17                          
           86405    12.5      85.47     832.12   brotli 6                         
           86880    12.6      19.16    2593.33   zstd 16                          
           87487    12.7      95.69     822.20   brotli 5                         
           87896    12.7      32.11    2653.18   zstd 15                          
           88136    12.8      38.44    2643.02   zstd 14                          
           88660    12.9      46.42    2622.92   zstd 13                          
           88977    12.9      88.06    2612.98   zstd 12                          
           89574    13.0     102.82    2573.98   zstd 11                          
           89632    13.0     114.42    2573.98   zstd 10                          
           90530    13.1     129.98    2526.84   zstd 9                           
           90683    13.1     138.94    2517.62   zstd 8                           
           91329    13.2     177.01    2454.90   zstd 7                           
           92976    13.5     201.76    2395.23   zstd 6                           
           93667    13.6     169.70     812.52   brotli 4                         
           95973    13.9     274.29    2261.73   zstd 5                           
           97817    14.2     247.52     739.36   brotli 3                         
          100575    14.6      47.21     731.52   zlib 9                           
          100734    14.6      68.52     728.43   zlib 8                           
          100852    14.6     302.16     708.24   brotli 2                         
          100857    14.6     430.60    2183.00   zstd 4                           
          101006    14.6     462.66    2183.00   zstd 3                           
          101445    14.7      94.57     721.58   zlib 7                           
          101965    14.8     105.49     719.32   zlib 6                           
          104262    15.1     126.43     700.33   zlib 5                           
          104733    15.2     664.57    2176.11   zstd 2                           
          106409    15.4     799.34    2239.70   zstd 1                           
          108348    15.7     140.09     678.96   zlib 4                           
          108871    15.8     531.45     711.90   brotli 1                         
          114394    16.6     186.29     700.33   zlib 3                           
          116565    16.9     195.97     689.14   zlib 2                           
          120313    17.4     200.41     669.09   zlib 1                           
          120596    17.5     683.67     636.37   brotli 0                         
    
    
    1,044,600 bytes - Nytimes homepage HTML:
    
          C Size  ratio%     C MB/s     D MB/s   Name
           50876     4.9       1.57    1245.05   brotli 11                        
           51256     4.9       4.42    1216.07   brotli 10                        
           56621     5.4       2.18    4145.24   zstd 22                          
           56906     5.4      55.26    1458.94   brotli 9                         
           57407     5.5       6.54    4229.15   zstd 21                          
           57554     5.5      80.17    1467.13   brotli 8                         
           57605     5.5       9.13    4229.15   zstd 19                          
           57636     5.5       7.16    4229.15   zstd 20                          
           57692     5.5     103.09    1460.98   brotli 7                         
           58232     5.6     137.85    1450.83   brotli 6                         
           58639     5.6       9.74    4229.15   zstd 18                          
           59114     5.7     155.59    1440.83   brotli 5                         
           59711     5.7      11.18    4389.08   zstd 17                          
           60221     5.8      11.48    4483.26   zstd 16                          
           60438     5.8      37.85    4522.08   zstd 15                          
           60678     5.8      49.99    4502.59   zstd 14                          
           60864     5.8     139.48    4483.26   zstd 12                          
           61250     5.9      70.25    4445.11   zstd 13                          
           61495     5.9     155.70    4445.11   zstd 11                          
           61587     5.9     174.48    4445.11   zstd 10                          
           62105     5.9     199.09    4352.50   zstd 9                           
           62207     6.0     213.36    4352.50   zstd 8                           
           62589     6.0     268.53    4298.77   zstd 7                           
           63581     6.1     309.51    4161.75   zstd 6                           
           64810     6.2     407.09    4048.84   zstd 5                           
           65682     6.3     277.67    1444.81   brotli 4                         
           66872     6.4     839.71    3987.02   zstd 4                           
           66891     6.4     947.05    4002.30   zstd 3                           
           69393     6.6     374.01    1396.52   brotli 3                         
           72221     6.9     458.36    1361.93   brotli 2                         
           72269     6.9    1396.52    3941.89   zstd 2                           
           72906     7.0    1440.83    4048.84   zstd 1                           
           76125     7.3     104.24    1122.02   zlib 9                           
           76128     7.3     111.57    1122.02   zlib 8                           
           76714     7.3     130.89    1118.42   zlib 7                           
           78781     7.5     142.98    1105.40   zlib 6                           
           80956     7.7     170.10    1084.74   zlib 5                           
           84011     8.0     941.93    1243.57   brotli 1                         
           84787     8.1     187.44    1059.43   zlib 4                           
           90873     8.7    1255.53    1267.72   brotli 0                         
           92904     8.9     296.85    1023.11   zlib 3                           
           95410     9.1     303.13    1007.33   zlib 2                           
          100473     9.6     306.51     976.26   zlib 1                           
    
    
    1,131,224 bytes - Dailymail homepage HTML:
    
          C Size  ratio%     C MB/s     D MB/s   Name
          125972    11.1       0.79     762.79   brotli 11                        
          128921    11.4       2.75     717.78   brotli 10                        
          139299    12.3       3.71    2371.54   zstd 22                          
          139527    12.3       4.19    2371.54   zstd 21                          
          139753    12.4      31.30     775.87   brotli 9                         
          140396    12.4      41.77     779.08   brotli 8                         
          140890    12.5      51.96     776.94   brotli 7                         
          140980    12.5       8.68    2361.64   zstd 19                          
          141706    12.5       6.94    2361.64   zstd 20                          
          142189    12.6      66.17     771.64   brotli 6                         
          142329    12.6       9.48    2337.24   zstd 18                          
          143021    12.6      10.64    2386.55   zstd 17                          
          143518    12.7      73.25     766.41   brotli 5                         
          144209    12.7      11.67    2469.92   zstd 16                          
          145692    12.9      24.00    2530.70   zstd 15                          
          146266    12.9      31.62    2508.26   zstd 14                          
          146801    13.0      37.29    2491.68   zstd 13                          
          148180    13.1      60.65    2480.75   zstd 12                          
          148926    13.2      73.36    2459.18   zstd 11                          
          149108    13.2      79.40    2459.18   zstd 10                          
          151002    13.3     112.55    2417.15   zstd 9                           
          151648    13.4     124.43    2406.86   zstd 8                           
          153777    13.6     152.29    2356.72   zstd 7                           
          157667    13.9     180.56    2285.30   zstd 6                           
          160375    14.2     161.67     906.43   brotli 4                         
          163821    14.5     245.44    2162.95   zstd 5                           
          164424    14.5     413.16    2205.12   zstd 4                           
          164850    14.6     438.97    2222.44   zstd 3                           
          169917    15.0     209.14     638.39   brotli 3                         
          171723    15.2      56.60     701.32   zlib 9                           
          171783    15.2      66.95     701.75   zlib 8                           
          172983    15.3      81.01     697.43   zlib 7                           
          173634    15.3     588.57    2209.42   zstd 2                           
          173645    15.4      85.71     695.71   zlib 6                           
          178068    15.7     248.40     622.24   brotli 2                         
          178646    15.8      99.56     679.01   zlib 5                           
          180946    16.0     782.85    2257.93   zstd 1                           
          184177    16.3     119.63     670.55   zlib 4                           
          185206    16.4     493.12     636.95   brotli 1                         
          195504    17.3     164.73     673.75   zlib 3                           
          198911    17.6     174.46     666.21   zlib 2                           
          201656    17.8     616.14     621.55   brotli 0                         
          205738    18.2     181.26     648.64   zlib 1
    Last edited by Sportman; 30th March 2018 at 02:51.

  27. Thanks:

    load (30th March 2018)

  28. #203
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,479
    Thanks
    26
    Thanked 122 Times in 96 Posts
    Sportman:
    Wrap the results in [code]content[/code] tags to improve rendering. E.g.
    Code:
    100,000,000 bytes - enwik8 HTML:
    
          C Size  ratio%     C MB/s     D MB/s   Name
        25380948    25.4       1.93     789.10   zstd 22                          
        25581278    25.6       2.07     797.84   zstd 21                          
        26028942    26.0       2.32     826.28   zstd 20                          
        27044781    27.0       0.67     544.84   brotli 11                        
        27549634    27.5       1.37     469.23   brotli 10                        
        27658726    27.7       3.59    1088.01   zstd 19                          
        28241804    28.2       4.63    1093.58   zstd 18                          
        28587027    28.6       5.15    1094.49   zstd 17                          
        29092452    29.1       7.19    1198.70   zstd 16                          
        29468810    29.5       8.48     562.08   brotli 9                         
        29774467    29.8       7.05    1134.91   zstd 15                          
        29806381    29.8      11.66     559.15   brotli 8                         
        30306203    30.3      16.62     549.14   brotli 7                         
        30336988    30.3      10.78    1116.89   zstd 14                          
        30444985    30.4      12.34    1112.50   zstd 13                          
        30991445    31.0      25.73     537.09   brotli 6                         
        31056443    31.1      17.32    1196.99   zstd 12                          
        31230227    31.2      24.85    1101.33   zstd 11                          
        31746267    31.7      32.66    1195.73   zstd 10                          
        31776975    31.8      34.97     518.82   brotli 5                         
        31992723    32.0      49.51    1198.77   zstd 9                           
        32257694    32.3      55.45    1097.16   zstd 8                           
        32737388    32.7      68.77    1055.72   zstd 7                           
        33304947    33.3      93.05    1036.07   zstd 6                           
        33515149    33.5      70.06     555.80   brotli 4                         
        34736348    34.7     123.32     990.54   zstd 5                           
        35205471    35.2     190.80    1027.37   zstd 4                           
        35687382    35.7     204.05    1048.35   zstd 3                           
        36475796    36.5      23.61     360.43   zlib 9                           
        36476038    36.5      23.67     360.44   zlib 8                           
        36498577    36.5      26.22     360.29   zlib 7                           
        36548925    36.5      29.31     359.81   zlib 6                           
        36685028    36.7     101.88     485.77   brotli 3                         
        36879096    36.9      40.76     355.71   zlib 5                           
        36936891    36.9     119.27     472.82   brotli 2                         
        37761110    37.8     254.71    1112.78   zstd 2                           
        37989516    38.0      60.38     358.60   zlib 4                           
        39123260    39.1     204.26     440.37   brotli 1                         
        39542367    39.5      61.71     371.88   zlib 3                           
        40746375    40.7      78.97     362.44   zlib 2                           
        40859471    40.9     371.56    1239.39   zstd 1                           
        41694594    41.7     264.04     406.38   brotli 0                         
        42298778    42.3      87.55     352.05   zlib 1

  29. #204
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    878
    Thanks
    80
    Thanked 315 Times in 219 Posts
    Quote Originally Posted by Piotr Tarsa View Post
    tags to improve rendering.
    Thanks, done.

  30. #205
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    749
    Thanks
    215
    Thanked 282 Times in 164 Posts
    Quote Originally Posted by Piotr Tarsa View Post
    Sportman:
    Code:
        25380948    25.4       1.93     789.10   zstd 22                          
        27044781    27.0       0.67     544.84   brotli 11
    Why compare unlimited window zstd window against limited window of brotli?

    Brotli should usually win density tests by 7.77 % when the same window size is used.
    Last edited by Jyrki Alakuijala; 30th March 2018 at 21:57.

  31. #206
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    565
    Thanks
    67
    Thanked 198 Times in 147 Posts
    I've adapted large window brotli in TurboBench Compression Benchmark (github only) and it is working without errors.
    enwik8 is now compressed to "24810184" bytes with "./turbobench brotli,11d29" (default is 24 bits = 16 mb windows).
    Binaries will be updated in the next days.

    P.S. You can generate an output formatted for encode.su (vBulletin format) with the command
    "./turbobench -p5 result_file.tbb"
    or
    "./turbobench -p5 -o result_file.tbb"

  32. Thanks:

    Jyrki Alakuijala (30th March 2018)

  33. #207
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    749
    Thanks
    215
    Thanked 282 Times in 164 Posts
    Quote Originally Posted by dnd View Post
    I've adapted large window brotli in TurboBench Compression Benchmark (github only) and it is working without errors.
    enwik8 is now compressed to "24810184" bytes with "./turbobench brotli,11d29" (default is 24 bits = 16 mb windows).
    Binaries will be updated in the next days.

    P.S. You can generate an output formatted for encode.su (vBulletin format) with the command
    "./turbobench -p5 result_file.tbb"
    or
    "./turbobench -p5 -o result_file.tbb"
    Yay! Didn't quite make the 7.77 % :-P

    RFC7932 and "br" content encoding requires a window smaller or equal to 24 bits, but "sbr" content encoding for shared brotli will allow a larger window to be specified. (((We haven't yet figured out the details around it, how to do that exactly in a way that still makes the system useful for simple mobile clients with limited resources, too.)))

  34. #208
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    878
    Thanks
    80
    Thanked 315 Times in 219 Posts
    Any plans to release a 1.0.5 Windows version as by version 1.0.4, 1.0.3, 1.0.0 etc? https://github.com/google/brotli/releases

  35. #209
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    239
    Thanks
    95
    Thanked 47 Times in 31 Posts
    Jyrki, say we wanted to store all relevant versions of jQuery and other libraries locally in the browser, instead of downloading them over and over. At minimum any CDN request for jQuery and other common libraries would be ignored and the relevant version of jQuery would be unpacked locally and near instantly (because CDN requests are for specific unmodified versions, and we would have those versions locally, whereas a website that hosted its own jQuery files might have modified them...)

    There's a huge amount of duplication from each library version to the next, so we wouldn't store them standalone, even compressed standalone. If we combined them somehow in a reversible way, brotli would be able to compress them very well, yes? You think a couple of MiB for 30 minified jQuery releases?

  36. #210
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    749
    Thanks
    215
    Thanked 282 Times in 164 Posts
    Quote Originally Posted by SolidComp View Post
    There's a huge amount of duplication from each library version to the next, ...
    Shared brotli is a possibility there.

Page 7 of 8 FirstFirst ... 5678 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •