Page 6 of 6 FirstFirst ... 456
Results 151 to 161 of 161

Thread: Google's compression projeсts

  1. #151
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    827
    Thanks
    236
    Thanked 297 Times in 178 Posts
    Quote Originally Posted by cssignet View Post
    i do like very much this kind of approach, where things would be 'simplified' for end-users make&keep it simple. regarding libs, sjpeg or libwebp (or LodePNG from Lode) are great example of this: they would be very easy to use/compile on most of OS/compiler, so trials could be done very quickly with just few lines. regarding UI, a tool should not overload targeted-users with useless stuff, but should be as simple as possible. finally and IMHO, some kind of automatic compression would be probably appreciated from modern tool
    In JPEG XL we decided to take as much out of the SIMD as we can and create progressive features that are more advanced than those of original JPEG. These decisions definitely do not play an advantage in simplicity and have given headaches with exposing SIMD-processing bugs in old compilers. For one clang, a two year old version (clang 6) doesn't work well for everything we want to do, but a 1.5 year old version (clang 7) already works.

    In JPEG XL simplicity comes in different forms than code: no specialized 8-bit mode, no yuv420 mode, each channel modeled as a float (instead of 8 or 10 bit integer), a psychovisual guarantee for loss to start happen everywhere in the image at the same time -- where some codecs start to destroy dark green and red/magenta areas first, one codec destroy brown areas first and then others, etc. Simplicity of use vs. simplicity of compiling. Simplicity of performance evaluations, too: other codecs publish performance eval in yuv420 mode with 8-bit dynamics, where we show them constantly at HDR/wide-gamut-ready yuv444/32 bit float dynamics.

    We tried to hide some of the increase in binary footprint, like JPEG XL uses brotli for metadata compression, which gives rather nice compression gain using technology that already sits in the browser, and we build the reference encoder using the same color management library that skia uses.

  2. #152
    Member
    Join Date
    Apr 2013
    Location
    France
    Posts
    45
    Thanks
    4
    Thanked 17 Times in 14 Posts
    Quote Originally Posted by myself
    also, i very quickly tried it regarding the lossless and it seems to be great
    those trials were about image data transformations, and it seems that the codec would be good enought on those
    Code:
    vxFkgPwD.png = 1099.39 KB
    
    > cjpegxl -q 100 vxFkgPwD.png vxFkgPwD.jxl
    Kernel  Time =     0.577 =   11%
    User    Time =    11.356 =  229%
    Process Time =    11.934 =  240%    Virtual  Memory =    528 MB
    Global  Time =     4.952 =  100%    Physical Memory =    400 MB <-- multithreaded
    vxFkgPwD.jxl = 865.66 KB
    perhaps somehow the transformation could be improved sometimes
    Code:
    akviv0R0.png = 1247.14 KB
    
    > cjpegxl -q 100 akviv0R0.png akviv0R0.jxl
    Kernel  Time =     0.358 =    9%
    User    Time =    11.091 =  278%
    Process Time =    11.450 =  287%    Virtual  Memory =    184 MB
    Global  Time =     3.986 =  100%    Physical Memory =    152 MB <-- multithreaded
    akviv0R0.jxl = 902.88 KB
    
    > cjpegxl -q 100 -s 9 akviv0R0.png akviv0R0-s9.jxl
    Kernel  Time =     0.468 =    9%
    User    Time =    15.678 =  302%
    Process Time =    16.146 =  311%    Virtual  Memory =    184 MB
    Global  Time =     5.178 =  100%    Physical Memory =    153 MB <-- multithreaded
    akviv0R0-s9.jxl = 892.17 KB
    
    > akviv0R0.png -> WebP (with prediction + pseudo-random colors re-ordering)
    Kernel  Time =     0.093 =    1%
    User    Time =     5.210 =   97%
    Process Time =     5.304 =   99%    Virtual  Memory =     81 MB
    Global  Time =     5.330 =  100%    Physical Memory =     78 MB <-- *not* multithreaded
    uSY86WmL.webp = 835.81 KB
    Quote Originally Posted by myself
    perhaps even more would be possible on paletted samples
    from end-users pov, some results could be unexpected atm, but maybe something went wrong on my side. still, could you confirm this result (from JPEG XL codec)?

    Code:
    yfgfInnU.png = 222.15 KB
    
    > cjpegxl -q 100 yfgfInnU.png yfgfInnU.jxl
    Kernel  Time =     0.312 =   12%
    User    Time =     5.413 =  216%
    Process Time =     5.725 =  228%    Virtual  Memory =    151 MB
    Global  Time =     2.502 =  100%    Physical Memory =    110 MB <-- multithreaded
    yfgfInnU.jxl = 260.82 KB
    
    > cjpegxl -q 100 -s 9 yfgfInnU.png yfgfInnU-s9.jxl
    Kernel  Time =     0.358 =   10%
    User    Time =     9.578 =  272%
    Process Time =     9.937 =  282%    Virtual  Memory =    153 MB
    Global  Time =     3.517 =  100%    Physical Memory =    112 MB <-- multithreaded
    yfgfInnU-s9.jxl = 257.59 KB
    
    > cwebp -lossless yfgfInnU.png -o yfgfInnU.webp
    Kernel  Time =     0.015 =    7%
    User    Time =     0.187 =   89%
    Process Time =     0.202 =   96%    Virtual  Memory =     25 MB
    Global  Time =     0.210 =  100%    Physical Memory =     24 MB <-- *not* multithreaded
    yfgfInnU.webp = 201.44 KB

  3. #153
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    827
    Thanks
    236
    Thanked 297 Times in 178 Posts
    Quote Originally Posted by cssignet View Post
    perhaps somehow the transformation could be improved sometimes
    I believe we have made some improvements in this area (simplicity and resource use of filters/predictors) and those will surface soon in the public repository.

    Encoder optimizations are on a rather low priority still, most optimizations are made for making decoding simpler and faster.

  4. #154
    Member
    Join Date
    Jun 2018
    Location
    Yugoslavia
    Posts
    54
    Thanks
    8
    Thanked 2 Times in 2 Posts
    I've tried lossless jpeg repack again with update jpegxl and some other image.
    looks normal now, it didn't skew it, but its not rotated so presumably something is lost (metadata?).
    is there some option for (d)encoder to unpack to original besides '-j'?

    I've tried zero-padding the start of unpacked file and comparing it to original, but couldn't find any match.
    also converted both to .bmp but they differ.

  5. Thanks:

    Jyrki Alakuijala (29th May 2020)

  6. #155
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    827
    Thanks
    236
    Thanked 297 Times in 178 Posts
    Could you file an issue either on jpegxl GitLab or on brunsli GitHub repo and we will look at it, and make them not differ. We have run this with lots of filed successfully so I doubt that this is either a special corner case or more likely a recent bug. We are currently converting these compressors into more streaming operation and to more easily streamable apis, and this bug might have come from that effort.

    Thank you in advance!

  7. #156
    Member
    Join Date
    Jun 2018
    Location
    Yugoslavia
    Posts
    54
    Thanks
    8
    Thanked 2 Times in 2 Posts
    ok Jyrki, will do.
    ​forgot to mention I don't have AVX CPU that was required before if that matters.

  8. #157
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    827
    Thanks
    236
    Thanked 297 Times in 178 Posts
    Quote Originally Posted by pklat View Post
    ok Jyrki, will do.
    ​forgot to mention I don't have AVX CPU that was required before if that matters.
    ​If we didn't fix that yet, we will fix it very soon now.

  9. #158
    Member
    Join Date
    Apr 2013
    Location
    France
    Posts
    45
    Thanks
    4
    Thanked 17 Times in 14 Posts
    Quote Originally Posted by Scope View Post
    Lossless Image Formats Comparison (Jpeg XL, AVIF, WebP, FLIF, PNG, ...)
    https://docs.google.com/spreadsheets...lqeDN2DuZ6yJ8/
    about the PNG tab, how did you measure speed?

  10. #159
    Member
    Join Date
    Nov 2019
    Location
    Moon
    Posts
    24
    Thanks
    5
    Thanked 26 Times in 14 Posts
    Quote Originally Posted by cssignet View Post
    about the PNG tab, how did you measure speed?
    I selected about ~900 Mb of PNG images (because the test of the whole set would take a lot of time and not all PNGs are processed correctly by all PNG optimizers and they skip them, improving their results) and several times measured the processing time of each optimizer on the CPU not loaded with other tasks.
    This is an average of all tests with parallel processing of 8 images on AVX2 CPU and 8 threads, 1x is the fastest optimization (ECT -1 compiled on GCC 10.1 with PGO, LTO, AVX2).

    Although it is not ready yet, but there are also some updates in comparison:
    - added ex-Jpeg rating, for more convenience in determining lossy images, but it doesn't work very well on non-photo images (and completely useless for PixelArt)
    - AVIF lossless was added and compression was performed on the latest version of Libavif 0.7.3 (aom:2.0.0, dav1d:0.7.0), as for YUV444, the speed was significantly increased, but the efficiency for static images has not changed much
    - updated WebP result (with recent changes in the main branch)
    - added comparison of the older Jpeg XL build (Speed 9) with the newer one (but comparison with the current build is not ready yet, because compression on Speed 8-9 takes a long time)

    I have tested near lossless modes for myself, but they are very difficult to add to a comparison without visual representation of the result (or any metrics).
    I also have a set with game screenshots (but the comparison is not ready yet).

    The set with non-RGB images hasn't been done yet, because I need enough free time to find, collect and convert such images, also it doesn't fit the current comparison, because I tried to use only non-converted public images, with a link to view each image (although it's possible to make a separate tab for such images).

    P.S. And about the need for lossless images in the Web, I do not think that they are completely useless for everything except UI-like, perhaps the need is much less for photographic images, but I see it more and more in Art images, comics/manga and game screenshots, especially given the ever-increasing speed of the Internet in the world
    Also all images in comparison are taken from very popular, viewed subreddits, I deliberately did not want to take my own images (because my needs may not reflect the needs of most other people).
    And considering the ineffectiveness of lossless mode in AVIF for RGB images, I hope that WebP v2 will have a more effective or separate lossless mode (as it was in WebP v1).

    Lossless Image Formats Comparison (Jpeg XL, AVIF, WebP, FLIF, PNG, ...) v0.5
    https://docs.google.com/spreadsheets...lqeDN2DuZ6yJ8/

    Total size bars on the chart do not need to be taken into account, they are not quite real, they are only for very rough representation
    Click image for larger version. 

Name:	PNG Optimization (total time spent).png 
Views:	28 
Size:	23.8 KB 
ID:	7718
    Last edited by Scope; Yesterday at 13:21.

  11. #160
    Member
    Join Date
    Apr 2013
    Location
    France
    Posts
    45
    Thanks
    4
    Thanked 17 Times in 14 Posts
    Quote Originally Posted by Scope
    ~900 Mb of PNG images (...) several times measured the processing time (...) average of all tests with parallel processing of 8 images on AVX2 CPU and 8 threads (...) ECT -1 compiled on GCC 10.1 with PGO, LTO, AVX2
    i would suggest a more simple, accurate, verifiable and *fair* test for time comparison. pingo/ECT binaries with same compiler/flags, cold-start running on dedicated resource (FX-4100 @ 3.6 Ghz — 8GB RAM — Windows 7 64-bit), tested on files found in PNG tab

    (aside note: i could not grab those
    https://i.redd.it/5lg9uz7fb7a41.png
    https://i.redd.it/6aiqgffywbk41.png
    https://i.redd.it/gxocab3x91e41.png
    https://i.redd.it/ks8z85usbg241.png
    https://i.redd.it/uuokrw18s4i41.png
    so input would be 1.89 GB (2 027 341 876 bytes) - 493 files)

    pingo (0.99 rc2 40) - ECT (f0b38f7 (0.8.3)) (with -strip)

    multi-processing (4x):
    Code:
    ECT -1 --mt-file
    Kernel  Time =    14.133 =    1%
    User    Time =  3177.709 =  390%
    Process Time =  3191.842 =  392%    Virtual  Memory =    438 MB
    Global  Time =   813.619 =  100%    Physical Memory =    433 MB
    
    
    pingo -s0
    Kernel  Time =    86.518 =   16%
    User    Time =  1740.393 =  328%
    Process Time =  1826.912 =  344%    Virtual  Memory =   1344 MB
    Global  Time =   530.104 =  100%    Physical Memory =   1212 MB
    
    
    ECT -5 --mt-file
    Kernel  Time =  1557.482 =   43%
    User    Time =  9361.869 =  259%
    Process Time = 10919.352 =  303%    Virtual  Memory =   1677 MB
    Global  Time =  3601.090 =  100%    Physical Memory =   1514 MB
    
    
    pingo -s5
    Kernel  Time =   144.550 =    6%
    User    Time =  6937.879 =  317%
    Process Time =  7082.429 =  324%    Virtual  Memory =   1378 MB
    Global  Time =  2183.105 =  100%    Physical Memory =   1193 MB
    file per file:

    Code:
    ECT -1
    Kernel  Time =    20.326 =    0%
    User    Time =  2963.472 =   93%
    Process Time =  2983.799 =   99%    Virtual  Memory =    283 MB
    Global  Time =  2984.405 =  100%    Physical Memory =    282 MB
    
    
    pingo -s0 -nomulti
    Kernel  Time =    68.468 =    4%
    User    Time =  1443.711 =   95%
    Process Time =  1512.180 =   99%    Virtual  Memory =    905 MB
    Global  Time =  1513.683 =  100%    Physical Memory =    887 MB
    
    
    ECT -5 --mt-deflate
    Kernel  Time =   886.538 =   14%
    User    Time =  8207.743 =  134%
    Process Time =  9094.281 =  149%    Virtual  Memory =   1000 MB
    Global  Time =  6083.433 =  100%    Physical Memory =    916 MB <-- multithreaded
    
    
    pingo -s5 -nomulti
    Kernel  Time =   109.107 =    1%
    User    Time =  5679.091 =   98%
    Process Time =  5788.198 =   99%    Virtual  Memory =    978 MB
    Global  Time =  5789.232 =  100%    Physical Memory =    980 MB <-- *not* multithreaded
    regular -sN profiles in pingo goes more for small/avg image size paletted/RGBA. if someone would seek for speed over space, -sN -faster could be used instead. on some samples, it could be still competitive

    https://i.redd.it/05vnjqzhrou31.png (13 266 623 bytes)
    Code:
    ECT -1 (out: 10 023 297 bytes)
    Kernel  Time =     0.140 =    2%
    User    Time =     5.928 =   97%
    Process Time =     6.068 =   99%    Virtual  Memory =     27 MB
    Global  Time =     6.093 =  100%    Physical Memory =     29 MB
    
    
    pingo -s0 (out: 8 777 351 bytes)
    Kernel  Time =     0.280 =    8%
    User    Time =     2.870 =   90%
    Process Time =     3.151 =   99%    Virtual  Memory =     98 MB
    Global  Time =     3.166 =  100%    Physical Memory =     90 MB
    
    
    pingo -s0 -faster (out: 9 439 005 bytes)
    Kernel  Time =     0.124 =    6%
    User    Time =     1.825 =   92%
    Process Time =     1.950 =   99%    Virtual  Memory =     86 MB
    Global  Time =     1.965 =  100%    Physical Memory =     78 MB

  12. #161
    Member
    Join Date
    Nov 2019
    Location
    Moon
    Posts
    24
    Thanks
    5
    Thanked 26 Times in 14 Posts
    Quote Originally Posted by cssignet View Post
    ​i would suggest a more simple, accurate, verifiable and *fair* test for time comparison. pingo/ECT binaries with same compiler/flags, cold-start running on dedicated resource (FX-4100 @ 3.6 Ghz — 8GB RAM — Windows 7 64-bit), tested on files found in PNG tab
    I can also add these results to the comparison (as tests on another configuration and number of threads) if they are done for all other modes and optimizers.
    That was my main problem, because not all optimizers processed correctly PNG from the whole set (and if they skip PNG after an error, they don't waste time optimizing it and it's not a very honest speed result).

    Otherwise, I also tried to make it simple, accurate and fair, the only difference is that:
    - I tested on another configuration, it's a dedicated i7-4770K (Haswell, AVX2, 16GB RAM, Windows 10 64-bit)
    - additional optimization flags were used during compilation (but they were used equally on all open source optimizers)
    - I ran tests 3 times on the same set with the same optimizers to make sure there was no impact of sudden Windows background processes, HDD caching, swap, etc. and get more accurate results
    - for a simpler and more convenient result, and since this is not the time spent on the whole set (for the reasons described above), for 1x was taken the fastest result, instead of a long numerical value of time spent such as Total Milliseconds - 2355561,4903.

    Mostly for the same reasons I didn't want to compare speed results at all, because they may depend on different configurations, CPU, even HDD speed (especially for fast modes), although they still give approximate data about speed of different optimizers.
    Quote Originally Posted by cssignet View Post
    (aside note: i could not grab those
    Yes, some files are deleted over time, this is one of the disadvantages of using public images, but my whole idea was to compare real public images rather than specialized test sets.
    Also, if needed, I can upload and provide a link (in pm) to a set of files on which I was comparing the speed.

Page 6 of 6 FirstFirst ... 456

Similar Threads

  1. RAISR - image compression with machine learning by Google
    By willvarfar in forum Data Compression
    Replies: 1
    Last Post: 13th January 2017, 19:59
  2. Google Chrome alert
    By Bulat Ziganshin in forum The Off-Topic Lounge
    Replies: 12
    Last Post: 3rd December 2016, 07:45
  3. Google released Snappy compression/decompression library
    By Sportman in forum Data Compression
    Replies: 11
    Last Post: 16th May 2011, 12:31
  4. Interested in Google-Wave?
    By Vacon in forum The Off-Topic Lounge
    Replies: 2
    Last Post: 29th November 2009, 19:11
  5. Did you know the google hashmap
    By thometal in forum Forum Archive
    Replies: 0
    Last Post: 4th February 2007, 15:21

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •