Page 8 of 8 FirstFirst ... 678
Results 211 to 224 of 224

Thread: Brotli

  1. #211
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    239
    Thanks
    95
    Thanked 47 Times in 31 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Shared brotli is a possibility there.
    What is that? A shared dictionary or schema like Protocol Buffers uses?

  2. #212
    Member SolidComp's Avatar
    Join Date
    Jun 2015
    Location
    USA
    Posts
    239
    Thanks
    95
    Thanked 47 Times in 31 Posts
    FYI, there have been several Brotli releases in the last few months: 1.0.3 - 1.0.6. It may perform better, so maybe some fresh benchmarks are in order. 1.0.5 was supposed to improve q=1 compression on small files, and the release notes for 1.0.6 are:


    • fix unaligned 64-bit accesses on AArch32
    • add missing files to the sources list
    • add ASAN/MSAN unaligned read specializations
    • fix CoverityScan "unused assignment" warning
    • fix JDK 8<->9 incompatibility
    • unbreak Travis builds
    • fix auto detect of bundled mode in cmake

  3. #213
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    330
    Thanks
    190
    Thanked 54 Times in 38 Posts
    v1.0.6

    Code:
    fix unaligned 64-bit accesses on AArch32
    add missing files to the sources list
    add ASAN/MSAN unaligned read specializations
    fix CoverityScan "unused assignment" warning
    fix JDK 8<->9 incompatibility
    unbreak Travis builds
    fix auto detect of bundled mode in cmake
    Please report any problems. Compiled with ICC15. GCC compiles are much slower.
    Attached Files Attached Files

  4. Thanks (3):

    hunman (30th September 2018),Sportman (7th January 2020),Stephan Busch (20th August 2019)

  5. #214
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    868
    Thanks
    80
    Thanked 314 Times in 218 Posts
    Quote Originally Posted by comp1 View Post
    Please report any problems.
    I get corrupt input error with Brotli 64-bit v1.0.6 and enwik9 decompress (compressed with -q 7 -w 22)

    A Brotli v1.07 Windows binary is also very welcome.

  6. #215
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,698
    Thanks
    270
    Thanked 1,182 Times in 653 Posts
    Compiled the source from repository.
    As usual, needs patches for IntelC/win.
    Attached Files Attached Files

  7. Thanks:

    Sportman (7th January 2020)

  8. #216
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    868
    Thanks
    80
    Thanked 314 Times in 218 Posts
    Input:
    1,000,000,000 bytes - enwik9

    Output:
    264,616,803 bytes

    brotli_gc82 -q 7 -w 22:

    1 core:
    50.330 2.005 RAMDISK
    50.821 2.063 NVMe
    50,585 2.035 SSD
    50,666 2.172 HD

    2+ cores:
    48.607 1.953 RAMDISK
    48.513 1.966 NVMe
    48.414 1.968 SSD
    48.498 2.100 HD

    brotli_ic19 -q 7 -w 22:

    1 core:
    53.445 2.235 RAMDISK
    53.386 2.208 NVMe
    53.570 2.228 SSD
    53.336 2.455 HD

    2+ cores:
    51.388 2.146 RAMDISK
    51.624 2.146 NVMe
    51.496 2.149 SSD
    51.662 2.178 HD

  9. #217
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    868
    Thanks
    80
    Thanked 314 Times in 218 Posts
    Input:
    10,000,000,000 bytes - enwik10

    Code:
    Output:
    3,770,151,519 bytes,     34.216 sec. - 26.236 sec., Brotli -0
    3,545,817,857 bytes,     42.569 sec. - 24.609 sec., Brotli -1
    3,268,318,786 bytes,     84.742 sec. - 22.775 sec., Brotli -2
    3,220,150,965 bytes,     94.157 sec. - 22.117 sec., Brotli -3
    3,095,248,795 bytes,    137.881 sec. - 20.738 sec., Brotli -4
    2,872,161,150 bytes,    251.839 sec. - 20.411 sec., Brotli -5
    2,780,438,122 bytes,    335.040 sec. - 19.575 sec., Brotli -6
    2,642,417,079 bytes,    603.066 sec. - 18.940 sec., Brotli -7
    2,573,497,114 bytes,  1,007.449 sec. - 18.956 sec., Brotli -8
    2,512,659,846 bytes,  1,734.988 sec. - 18.909 sec., Brotli -9
    2,220,027,943 bytes,  7,439.064 sec. - 22.690 sec., Brotli -10
    2,172,589,967 bytes, 14,232.804 sec. - 20.310 sec., Brotli -11

  10. #218
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    749
    Thanks
    215
    Thanked 281 Times in 164 Posts
    Brotli 10 and 11 should be relatively strong on this if enwik10 contains Japanese and other utf-8 heavy languages. If not, we need to take a look at the hashing...

  11. #219
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    155
    Thanks
    15
    Thanked 17 Times in 15 Posts
    Quote Originally Posted by Sportman View Post
    Input:
    10,000,000,000 bytes - enwik10

    Code:
    Output:
    3,770,151,519 bytes,     34.216 sec. - 26.236 sec., Brotli -0
    3,545,817,857 bytes,     42.569 sec. - 24.609 sec., Brotli -1
    3,268,318,786 bytes,     84.742 sec. - 22.775 sec., Brotli -2
    3,220,150,965 bytes,     94.157 sec. - 22.117 sec., Brotli -3
    3,095,248,795 bytes,    137.881 sec. - 20.738 sec., Brotli -4
    2,872,161,150 bytes,    251.839 sec. - 20.411 sec., Brotli -5
    2,780,438,122 bytes,    335.040 sec. - 19.575 sec., Brotli -6
    2,642,417,079 bytes,    603.066 sec. - 18.940 sec., Brotli -7
    2,573,497,114 bytes,  1,007.449 sec. - 18.956 sec., Brotli -8
    2,512,659,846 bytes,  1,734.988 sec. - 18.909 sec., Brotli -9
    2,220,027,943 bytes,  7,439.064 sec. - 22.690 sec., Brotli -10
    2,172,589,967 bytes, 14,232.804 sec. - 20.310 sec., Brotli -11
    Could you compare with crush v1.4 ??

  12. #220
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    565
    Thanks
    67
    Thanked 198 Times in 147 Posts
    Can you please post the full command lines used for each level and the I/O devices (HD,NVM?).

  13. #221
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    868
    Thanks
    80
    Thanked 314 Times in 218 Posts
    Quote Originally Posted by dnd View Post
    Can you please post the full command lines used for each level and the I/O devices (HD,NVM?).
    I created and started a batch file bro.bat see bro.zip
    NVMe used.
    Attached Files Attached Files

  14. #222
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    565
    Thanks
    67
    Thanked 198 Times in 147 Posts
    Better use the "large_window=30" option with quality 10 and 11.

  15. #223
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    868
    Thanks
    80
    Thanked 314 Times in 218 Posts
    Quote Originally Posted by dnd View Post
    Better use the "large_window=30" option with quality 10 and 11.
    I start first with default options.
    If I use it, then I write it, like here https://encode.su/threads/2119-Zstan...ll=1#post62944

  16. #224
    Member
    Join Date
    Mar 2013
    Location
    Worldwide
    Posts
    565
    Thanks
    67
    Thanked 198 Times in 147 Posts
    TurboBench is using large window brotli with q 10 and 11 per default.
    Comparison with other compressors is not possible with 8/16 MB window when you don't specify the "large_window" option.

Page 8 of 8 FirstFirst ... 678

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •