Activity Stream

Filter
Sort By Time Show
Recent Recent Popular Popular Anytime Anytime Last 24 Hours Last 24 Hours Last 7 Days Last 7 Days Last 30 Days Last 30 Days All All Photos Photos Forum Forums
  • Jyrki Alakuijala's Avatar
    Today, 10:17
    Jyrki Alakuijala replied to a thread Brotli in Data Compression
    Based on a quick look at the makefiles, we are not using the fast math option. However, there can be more uncertainty, like perhaps using multiply-and-add as a single instruction leading to a different result that doing multiply and add as two separate instructions. (I'm a bit out of touch with this field. Compilers, vectorization and new instructions are improved constantly.)
    249 replies | 81707 view(s)
  • Kirr's Avatar
    Today, 05:58
    Kirr replied to a thread Zstandard in Data Compression
    From source, when possible. Thanks, will clarify it on website (in the next update).
    428 replies | 130032 view(s)
  • SolidComp's Avatar
    Today, 05:16
    SolidComp replied to a thread Zstandard in Data Compression
    Do you build the compressors from source, or do you use the builds provided by the projects?
    428 replies | 130032 view(s)
  • SolidComp's Avatar
    Today, 05:11
    SolidComp replied to a thread Brotli in Data Compression
    Do you specify fast math in the makefile or cmake?
    249 replies | 81707 view(s)
  • SolidComp's Avatar
    Today, 05:11
    SolidComp replied to a thread Brotli in Data Compression
    Where's the dictionary?
    249 replies | 81707 view(s)
  • Kirr's Avatar
    Today, 02:59
    Kirr replied to a thread Zstandard in Data Compression
    zstd is now updated to 1.4.5 in my benchmark: http://kirr.dyndns.org/sequence-compression-benchmark/ I noticed good improvement in decompression speed for all levels, and some improvement in compression speed for slower levels. (Though I am updating from 1.4.0, so the improvement may be larger than from 1.4.4).
    428 replies | 130032 view(s)
  • redrabbit's Avatar
    Today, 02:09
    Thanks for the explanation and the testing
    84 replies | 13005 view(s)
  • terrelln's Avatar
    Today, 01:55
    terrelln replied to a thread Zstandard in Data Compression
    Both single-thread and multi-thread modes are deterministic, but they produce different results. Multi-threaded compression produces the same output with any number of threads. The zstd cli defaults to multi-threaded compression with 1 worker thread. You can opt into single-thread compression with --single-thread.
    428 replies | 130032 view(s)
  • schnaader's Avatar
    Yesterday, 23:47
    OK, so here's the long answer. I could reproduce the bad performance of the Life is Strange 2 testfile, my results are in the table below. There are two things this all boils down to: preflate (vs. zlib brute force in xtool and Precomp 0.4.6) and multithreading. Note that both the Life is Strange 2 times and the decompressed size are very similar for Precomp 0.4.6 and xtool when considering the multithreading factor (computed by using the time command and dividing "user" time by "real" time). Also note that the testfile has many small streams (64 KB decompressed each), preflate doesn't seem to use its multithreading in that case. Although preflate can be slower than zlib brute force, it also has big advantages which can be seen when looking at the Eternal Castle testfile. It consists of big PNG files, preflate can make use of multithreading (though not fully utilizing all cores) and is faster than the zlib brute force. And the zlib brute force doesn't even manage to recompress any of the PNG files. Xtool's (using reflate) decompressed size is somewhere between those two, most likely because reflate doesn't parse multi PNG files and can only decompress parts of them because of this. So, enough explanation, how can the problem be solved? Multithreading, of course. The current branch already features multithreading for JPEG when using -r and I'm working on it for deflate streams. When it's done, I'll post fresh results for the Life is Strange 2 testfile, should be very close to xtool if things work out well. Multithreaded -cn or -cl though is a bit more complex, I've got some ideas, but have to test them and it will take longer. Test system: Hetzner vServer CPX21: AMD Epyc, 3 cores @ 2.5 GHz, Ubuntu 20.04 64-Bit Eternal Castle testfile, 223,699,564 bytes program decompressed size time (decompression/recompression) multithreading factor (decompression/recompression) compressed size (-nl) --- Precomp 0.4.8dev -cn -d0 5,179,454,907 5 min 31 s / 4 min 45 s 1.73 / 1.64 118,917,128 Precomp 0.4.6 -cn -d0 223,699,589 8 min 31 s 1.00 173,364,804 xtool (redrabbit's result) 2,099,419,005 Life is Strange 2 testfile, 632,785,771 bytes program decompressed size time (decompression/recompression) multithreading factor (decompression/recompression) --- Precomp 0.4.8dev -cn -intense0 -d0 1,499,226,364 3 min 21 s / 2 min 14 s 0.91 / 0.99 Precomp 0.4.8dev (after tempfile fix) 1,499,226,364 3 min 11 s / 2 min 21 s 0.92 / 0.99 Precomp 0.4.6 -cn -intense0 -d0 1,497,904,244 1 min 55 s / 1 min 43 s 0.93 / 0.98 xtool 0.9 e:precomp:32mb,t4:zlib (Wine) 1,497,672,882 46 s / 36 s 2.75 / 2.87
    84 replies | 13005 view(s)
  • Jyrki Alakuijala's Avatar
    Yesterday, 23:36
    Jyrki Alakuijala replied to a thread Brotli in Data Compression
    I don't remember what was improved. Perhaps hashing. There is a lot of floating point in brotli 10 and 11. I don't know how compiler invariant it is the way we do it. I'd assume it to be well-defined, but this could be a naive position.
    249 replies | 81707 view(s)
  • Shelwien's Avatar
    Yesterday, 21:20
    Shelwien replied to a thread Brotli in Data Compression
    Normally it just needs more data for training. But here I made a workaround for you:
    249 replies | 81707 view(s)
  • schnaader's Avatar
    Yesterday, 20:44
    @redrabbit: Just a quick post to let you know I'm currently in researching the bad performance you describe on those 2 files and stuff I found out while profiling the latest Precomp version. Don't waste too much time on this. As I saw your post, I wondered about where zlib is still used as preflate is doing all deflate related stuff itself. But zlib is indeed still used to speed up intense and brute mode (testing the first few bytes of a potential stream to avoid recompressing false positives). But profiling the latest version shows that for the Life Is Strange 2 file you posted, this is only using 0.3% of CPU time (of .pak -> .pcf, in -r zlib isn't used at all). So using a faster zlib library could only speed up things by 0.3%. On the other hand, I found something else and fixed it some minutes ago in both branches: About 5% of CPU time was wasted because uncompressed data was written to a file to prepare recursion even though both "-intense0" and "-d0" disable recursion, so the temporary file wasn't used at all. Fixed this by writing the file only if it's used. Testing this shows it works: 3 min 11 s instead of 3 min 21 s for "-cn -intense0 -d0" of the Life Is Strange 2 file. Not much, but some progress. Might have more impact on non-SSD drives.
    84 replies | 13005 view(s)
  • SolidComp's Avatar
    Yesterday, 20:36
    SolidComp replied to a thread Brotli in Data Compression
    I tried to create a dictionary with --train, but I get an error saying not enough samples or something. I tried it with just the jQuery file as the training sample, which used to work in the past. Then I tried two, then three, then four jQuery files (the previous versions, 3.5.0, 3.4.1, etc.), and still get the error even with four files. Not sure what I'm doing wrong.
    249 replies | 81707 view(s)
  • Hakan Abbas's Avatar
    Yesterday, 19:59
    In data compression, working speed is as important as compression rate. When looking at the sector, it is clearly seen that faster products are preferred even if they are low in efficiency. It is not preferable to spend much more energy than necessary for a small data saving. No matter who is behind the product. In most cases, while performing the calculations, the cost (cpu, ram ...) of these transactions is taken into consideration. Whenever possible, situations requiring excessive processing load are avoided. However, as we have seen, it cannot be said that attention is paid to these points for AVIF/AV1.
    14 replies | 508 view(s)
  • Shelwien's Avatar
    Yesterday, 19:58
    Shelwien replied to a thread Brotli in Data Compression
    Its unfair comparison, since both brotli and zstd have support for external dictionary, but brotli silently uses integrated one 89,476 jquery-3.5.1.min.js 27,959 jquery-3.5.1.min.bro // brotli_gc82.exe -q 11 -fo jquery-3.5.1.min.bro jquery-3.5.1.min.js 28.754 jquery-3.5.1.min.bro // brotli with dictionary zeroed out in .exe 29,453 jquery-3.5.1.min.zst // zstd.exe --ultra -22 -fo jquery-3.5.1.min.zst jquery-3.5.1.min.js 28,942 jquery-3.5.1.min.zst // zstd.exe --ultra -22 -fo jquery-3.5.1.min.zst -D dictionary.bin jquery-3.5.1.min.js This example uses brotli's default dictionary for zstd, but we can generate a custom dictionary for zstd, while it is harder to do for brotli. Yes, brotli at max settings has stronger entropy model than zstd. But it also has 5x slower encoding and 2x slower decoding. And actual compression difference is still uncertain, since we can build a larger specialized dictionary for target data with zstd --train.
    249 replies | 81707 view(s)
  • Jyrki Alakuijala's Avatar
    Yesterday, 19:40
    My basic understanding is that avif decoders are roughly half the speed of jpeg xl decoders. Getting the fastest avif decoding may require turning off some features that are always on in jpeg xl, for example no yuv444, no more than 8 bit of dynamics. It may be that hardware decoders for avif may not be able to do streaming, i.e., to display that part of the image that is already decoded. For some applications this can be a blocker.
    14 replies | 508 view(s)
  • Jyrki Alakuijala's Avatar
    Yesterday, 19:30
    Pik is a superfast simple format for photography level qualities (1.5+ bpp), and gives great quality/density there. It used dct8x8. The requirements for JPEG XL included lower rates, down to 0.06 bpp was discussed. Variable sizes of dcts and filtering improve the performance at lower bpps, and keep it at state-of-the art down to 0.4 bpp and somewhat decent at 0.15 bpp. This was adding a lot of code and 2x decoding slowdown to cover a larger bpp range. Further, we integrated FUIF into PIK. We are still in the process of figuring out all the possibilities it brings. FUIF seems much less psycho usually efficient, but more versatile coding. Luca and Jon developed FUIF code further after integrating it into JPEG XL. Jon has been in general great to collaborate with and I am quite proud for having made the initial proposals of basing JPEG XL on these two codecs. Everyone in the team has grown very comfortable with the fusion of these two codecs.
    14 replies | 508 view(s)
  • Jyrki Alakuijala's Avatar
    Yesterday, 19:16
    I'm not an expert on jpeg xt. Xt is likely a HDR patch on usual jpegs. Not particularly effective at compression density.
    14 replies | 508 view(s)
  • LucaBiondi's Avatar
    Yesterday, 19:07
    LucaBiondi replied to a thread Paq8pxd dict in Data Compression
    Thank you!!
    913 replies | 312843 view(s)
  • DZgas's Avatar
    Yesterday, 18:36
    DZgas replied to a thread JPEG XL vs. AVIF in Data Compression
    I think - Large corporations did AV1 codec for self, not for users haha. But decoding with the standard Dav1d is normal...if fact that my laptop can't VP9 1080p and AV1 720p and on the verge can HEVC 1080p. Problems of progress - I am slow. Obviously AV1 is slowest codec and the strongest of all. Encoding speeds are currently killing him. Despite the fact that it is ~ 25% better than HEVC or VP9, ​​it is 10-20 times slower, this is serious for people who do not have powerful PC/Server. Well, almost the Internet is still using the old AVC, because it’s fast and you can decode on Watch.
    14 replies | 508 view(s)
  • SolidComp's Avatar
    Yesterday, 17:25
    SolidComp replied to a thread Zstandard in Data Compression
    Sportman, why is the compressed size different for single thread vs multithreaded? Is it supposed to produce different results? I thought it would be deterministic at any given compression level.
    428 replies | 130032 view(s)
  • SolidComp's Avatar
    Yesterday, 17:19
    Yes, AVIF was not intended for cameras. The encoders are still incredibly slow, as are the encoders for AV1 video (though I think Intel's SVT AV1 encoder is improving). Do you know if the decoders are reasonably fast?
    14 replies | 508 view(s)
  • SolidComp's Avatar
    Yesterday, 17:18
    Jyrki, is either JPEG XL or AVIF related to PIK? What became of PIK? And I'm confused by JPEG XT – do you know if it's related to XL?
    14 replies | 508 view(s)
  • SolidComp's Avatar
    Yesterday, 17:02
    SolidComp replied to a thread Brotli in Data Compression
    Hi all – I'm impressed with the results of compressing jQuery with brotli, compared to Zstd and libdeflate (gzip): Original jQuery 3.5.1 (latest): 89,476 bytes (this is the minified production version from jQuery.com: Link) libdeflate 1.6 gzip -11: 36,043 (libdeflate adds two extra compression levels to zlib gzip's nine) Zstd 1.4.5 -22: 29,453 brotli 1.0.4 -11: 28,007 brotli 1.0.7 -11: 27,954 Update: 7-Zip's gzipper is incredible: 29,960 bytes. I'm not sure why it's so much better than libdeflate, or how it's so close to Zstd and brotli. Compression of web files is much more important to me than the weird benchmarks that are typically used. And this is where brotli shines, not surprisingly since it was designed for the web. Note that brotli has a dictionary, generated from web files, whereas Zstd and libdeflate do not. You can generate a dictionary with Zstd, but it keeps giving me an error saying there aren't enough samples... Brotli 1.0.7 performs slightly better than 1.0.4, which was surprising since there was nothing in the release notes that indicated improvements for the max compression setting (-11), just an improvement for the -1 setting. The only other difference is that I compiled my 1.0.7 version myself in Visual Studio 2019, dynamically linked, whereas my 1.0.4 version is the official build from GitHub, a static executable (1.0.4 is the last version they released builds for – all they've released is source code for 1.0.5 through 1.0.7). Jyrki, should compression results (size) be exactly the same across compilers, deterministic given the same settings? So it has to be something in the source code of 1.0.7 compared to 1.0.4 that explains the improvement, right, not Visual Studio?
    249 replies | 81707 view(s)
  • moisesmcardona's Avatar
    Yesterday, 16:30
    The commits are there. Kaitz just didn't posted the compiled versions: v87: https://github.com/kaitz/paq8pxd/commit/86969e4174f8f3f801f9a0d94d36a8cbda783961 v88: https://github.com/kaitz/paq8pxd/commit/7969cc107116c31cd997f37359b433994fea1f6d I've attached them with the source from their respective commits. Compiled with march=native on my AMD Ryzen 9 CPU.
    913 replies | 312843 view(s)
  • Jyrki Alakuijala's Avatar
    Yesterday, 00:19
    Jpeg XL is not frozen yet. (Other than the jpeg repacking part of it.) Our freezing schedule is end of August 2020. Before that it is not a good idea to integrate for other than testing use.
    14 replies | 508 view(s)
  • Darek's Avatar
    23rd May 2020, 20:54
    Darek replied to a thread Paq8pxd dict in Data Compression
    Scores of my testset for paq8pxd_v89. Slight reverse in total testset. Mixed scores for textual files, worse scores for exe files.
    913 replies | 312843 view(s)
  • schnaader's Avatar
    23rd May 2020, 20:16
    You can try to reverse engineer the code coming from this C compiler: https://github.com/Battelle/movfuscator Will be very hard as the binary will only contain mov instructions and has some (optional?) randomization applied, IIRC. On the other hand, performance of the programs compiled with this won't be very good. There's an example of DOOM 1 compiled this way that renders a frame around every 7 hours. https://github.com/xoreaxeaxeax/movfuscator/tree/master/validation/doom
    5 replies | 803 view(s)
  • Jyrki Alakuijala's Avatar
    23rd May 2020, 20:02
    I believe we have made some improvements in this area (simplicity and resource use of filters/predictors) and those will surface soon in the public repository. Encoder optimizations are on a rather low priority still, most optimizations are made for making decoding simpler and faster.
    152 replies | 35452 view(s)
  • Shelwien's Avatar
    23rd May 2020, 18:08
    1) This contest is intended for more practical algorithms - lowest allowed speed would be something like 250kb/s. So no PAQs,NN/ML most likely. 2) Archiver size can be pretty large - precomp+zstd could be 3Mb easily, more if there're several compression methods. 3) Processing speed would be a part of ranking, so dictionary preprocessing won't be a free way to decrease compressed size.
    8 replies | 544 view(s)
  • cssignet's Avatar
    23rd May 2020, 16:45
    those trials were about image data transformations, and it seems that the codec would be good enought on those vxFkgPwD.png = 1099.39 KB > cjpegxl -q 100 vxFkgPwD.png vxFkgPwD.jxl Kernel Time = 0.577 = 11% User Time = 11.356 = 229% Process Time = 11.934 = 240% Virtual Memory = 528 MB Global Time = 4.952 = 100% Physical Memory = 400 MB <-- multithreaded vxFkgPwD.jxl = 865.66 KB perhaps somehow the transformation could be improved sometimes akviv0R0.png = 1247.14 KB > cjpegxl -q 100 akviv0R0.png akviv0R0.jxl Kernel Time = 0.358 = 9% User Time = 11.091 = 278% Process Time = 11.450 = 287% Virtual Memory = 184 MB Global Time = 3.986 = 100% Physical Memory = 152 MB <-- multithreaded akviv0R0.jxl = 902.88 KB > cjpegxl -q 100 -s 9 akviv0R0.png akviv0R0-s9.jxl Kernel Time = 0.468 = 9% User Time = 15.678 = 302% Process Time = 16.146 = 311% Virtual Memory = 184 MB Global Time = 5.178 = 100% Physical Memory = 153 MB <-- multithreaded akviv0R0-s9.jxl = 892.17 KB > akviv0R0.png -> WebP (with prediction + pseudo-random colors re-ordering) Kernel Time = 0.093 = 1% User Time = 5.210 = 97% Process Time = 5.304 = 99% Virtual Memory = 81 MB Global Time = 5.330 = 100% Physical Memory = 78 MB <-- *not* multithreaded uSY86WmL.webp = 835.81 KB from end-users pov, some results could be unexpected atm, but maybe something went wrong on my side. still, could you confirm this result (from JPEG XL codec)? yfgfInnU.png = 222.15 KB > cjpegxl -q 100 yfgfInnU.png yfgfInnU.jxl Kernel Time = 0.312 = 12% User Time = 5.413 = 216% Process Time = 5.725 = 228% Virtual Memory = 151 MB Global Time = 2.502 = 100% Physical Memory = 110 MB <-- multithreaded yfgfInnU.jxl = 260.82 KB > cjpegxl -q 100 -s 9 yfgfInnU.png yfgfInnU-s9.jxl Kernel Time = 0.358 = 10% User Time = 9.578 = 272% Process Time = 9.937 = 282% Virtual Memory = 153 MB Global Time = 3.517 = 100% Physical Memory = 112 MB <-- multithreaded yfgfInnU-s9.jxl = 257.59 KB > cwebp -lossless yfgfInnU.png -o yfgfInnU.webp Kernel Time = 0.015 = 7% User Time = 0.187 = 89% Process Time = 0.202 = 96% Virtual Memory = 25 MB Global Time = 0.210 = 100% Physical Memory = 24 MB <-- *not* multithreaded yfgfInnU.webp = 201.44 KB
    152 replies | 35452 view(s)
  • DZgas's Avatar
    23rd May 2020, 16:18
    DZgas replied to a thread JPEG XL vs. AVIF in Data Compression
    Of course JpegXR is not JpegXL but can definitely say that AVIF will not be used in real time on cameras, it is Slowest. AVIF is developed by Alliance for Open Media, which includes most large corporations. AVIF is well suited for everything on the Internet, photographs of articles, video previews, stickers, and everything else. JpegXR has supported by Microsoft. You can view in the standard Windows photo viewer. And JpegXL?...was released...But I think - no one noticed it. There are currently no user-friendly programs for encoding and viewing JpegXL.
    14 replies | 508 view(s)
  • ivan2k2's Avatar
    23rd May 2020, 15:21
    ivan2k2 replied to a thread Zstandard in Data Compression
    error, if files in filelist separated by windows style new line (0x0d,0x0a): Error : util.c, 283 : fgets(buf, (int) len, file) unix style new line works fine.
    428 replies | 130032 view(s)
  • mhajicek's Avatar
    23rd May 2020, 14:09
    I d say private data set sounds better, cause then there is a higher chance the resulting compressor/algorithm will have some more general use, not just something designed for one file, otherwise pretty useless. So id say it supports usefull development more. Maybe the decompressor size could be limited even more, 16MB is really a lot - well, that depends on whether dictionary and other help data was just a considered side effect of the option 2], or if it was intended to encourage ppl to use that in development :)
    8 replies | 544 view(s)
  • Sportman's Avatar
    23rd May 2020, 12:50
    Sportman replied to a thread Zstandard in Data Compression
    enwik10: 3,638,532,709 bytes, 23.290 sec. - 10.649 sec., zstd -1 --ultra (v1.4.5) 3,325,793,817 bytes, 32.959 sec. - 11.632 sec., zstd -2 --ultra (v1.4.5) 3,137,188,839 bytes, 42.442 sec. - 11.994 sec., zstd -3 --ultra (v1.4.5) 3,072,048,223 bytes, 44.923 sec. - 12.828 sec., zstd -4 --ultra (v1.4.5) 2,993,531,459 bytes, 72.322 sec. - 12.827 sec., zstd -5 --ultra (v1.4.5) 2,921,997,106 bytes, 95.852 sec. - 12.613 sec., zstd -6 --ultra (v1.4.5) 2,819,369,488 bytes, 132.442 sec. - 11.922 sec., zstd -7 --ultra (v1.4.5) 2,780,718,316 bytes, 168.737 sec. - 11.724 sec., zstd -8 --ultra (v1.4.5) 2,750,214,835 bytes, 237.175 sec. - 11.574 sec., zstd -9 --ultra (v1.4.5) 2,694,582,971 bytes, 283.778 sec. - 11.564 sec., zstd -10 --ultra (v1.4.5) 2,669,751,039 bytes, 355.330 sec. - 11.651 sec., zstd -11 --ultra (v1.4.5) 2,645,099,063 bytes, 539.770 sec. - 11.658 sec., zstd -12 --ultra (v1.4.5) 2,614,435,940 bytes, 717.361 sec. - 11.766 sec., zstd -13 --ultra (v1.4.5) 2,569,453,043 bytes, 894.063 sec. - 11.872 sec., zstd -14 --ultra (v1.4.5) 2,539,608,782 bytes, 1,198.939 sec. - 11.795 sec., zstd -15 --ultra (v1.4.5) 2,450,374,607 bytes, 1,397.298 sec. - 11.547 sec., zstd -16 --ultra (v1.4.5) 2,372,309,135 bytes, 1,994.123 sec. - 11.414 sec., zstd -17 --ultra (v1.4.5) 2,339,536,175 bytes, 2,401.207 sec. - 11.819 sec., zstd -18 --ultra (v1.4.5) 2,299,200,392 bytes, 3,093.583 sec. - 12.295 sec., zstd -19 --ultra (v1.4.5) 2,196,998,753 bytes, 3,838.985 sec. - 12.952 sec., zstd -20 --ultra (v1.4.5) 2,136,031,972 bytes, 4,488.867 sec. - 13.171 sec., zstd -21 --ultra (v1.4.5) 2,079,998,491 bytes, 5,129.788 sec. - 12.915 sec., zstd -22 --ultra (v1.4.5) enwik10: 3,642,089,943 bytes, 28.752 sec. - 10.717 sec., zstd -1 --ultra --single-thread (v1.4.5) 3,336,007,957 bytes, 38.991 sec. - 11.808 sec., zstd -2 --ultra --single-thread (v1.4.5) 3,133,763,440 bytes, 48.671 sec. - 12.157 sec., zstd -3 --ultra --single-thread (v1.4.5) 3,065,081,662 bytes, 50.724 sec. - 12.904 sec., zstd -4 --ultra --single-thread (v1.4.5) 2,988,125,022 bytes, 79.664 sec. - 13.073 sec., zstd -5 --ultra --single-thread (v1.4.5) 2,915,934,603 bytes, 103.971 sec. - 12.798 sec., zstd -6 --ultra --single-thread (v1.4.5) 2,811,448,067 bytes, 148.300 sec. - 12.125 sec., zstd -7 --ultra --single-thread (v1.4.5) 2,775,621,897 bytes, 188.946 sec. - 11.804 sec., zstd -8 --ultra --single-thread (v1.4.5) 2,744,751,362 bytes, 255.285 sec. - 11.929 sec., zstd -9 --ultra --single-thread (v1.4.5) 2,690,272,721 bytes, 304.380 sec. - 11.737 sec., zstd -10 --ultra --single-thread (v1.4.5) 2,663,964,945 bytes, 380.876 sec. - 11.848 sec., zstd -11 --ultra --single-thread (v1.4.5) 2,639,230,515 bytes, 561.791 sec. - 11.774 sec., zstd -12 --ultra --single-thread (v1.4.5) 2,609,728,690 bytes, 705.747 sec. - 11.646 sec., zstd -13 --ultra --single-thread (v1.4.5) 2,561,381,234 bytes, 896.689 sec. - 11.777 sec., zstd -14 --ultra --single-thread (v1.4.5) 2,527,193,467 bytes, 1,227.455 sec. - 11.893 sec., zstd -15 --ultra --single-thread (v1.4.5) 2,447,614,045 bytes, 1,360.777 sec. - 11.614 sec., zstd -16 --ultra --single-thread (v1.4.5) 2,370,639,588 bytes, 1,953.282 sec. - 11.641 sec., zstd -17 --ultra --single-thread (v1.4.5) 2,337,506,087 bytes, 2,411.038 sec. - 11.971 sec., zstd -18 --ultra --single-thread (v1.4.5) 2,299,225,559 bytes, 2,889.098 sec. - 12.184 sec., zstd -19 --ultra --single-thread (v1.4.5) 2,197,171,322 bytes, 3,477.477 sec. - 12.862 sec., zstd -20 --ultra --single-thread (v1.4.5) 2,136,340,302 bytes, 4,024.675 sec. - 12.940 sec., zstd -21 --ultra --single-thread (v1.4.5) 2,080,479,075 bytes, 4,568.550 sec. - 12.934 sec., zstd -22 --ultra --single-thread (v1.4.5) Difference zstd x --ultra --single-thread (v1.4.4) versus (v1.4.5): zstd -1, 0.168 sec., -0.137 sec., -501 bytes zstd -2, 0.389 sec., -0.610 sec., -87 bytes zstd -3, 0.021 sec., -0.490 sec., 0 bytes zstd -4, 0.029 sec., -0.441 sec., 2 bytes zstd -5, 1.127 sec., -0.204 sec., 0 bytes zstd -6, 1.427 sec., -0.504 sec., 0 bytes zstd -7, 4.892 sec., -0.257 sec., 0 bytes zstd -8, 2.735 sec., -0.202 sec., 0 bytes zstd -9, 2.494 sec., -0.008 sec., 0 bytes zstd -10, 3.077 sec., -0.181 sec., 0 bytes zstd -11, 3.802 sec., -0.165 sec., 0 bytes zstd -12, 0.884 sec., -0.158 sec., 0 bytes zstd -13, 0.184 sec., -0.146 sec., 0 bytes zstd -14, -2.345 sec., -0.114 sec., 0 bytes zstd -15, -3.486 sec., -0.124 sec., 0 bytes zstd -16, -3.167 sec., -0.102 sec., 0 bytes zstd -17, 15.255 sec., -0.041 sec., 0 bytes zstd -18, 39.073 sec., -0.147 sec., 0 bytes zstd -19, 16.304 sec., -0.151 sec., 0 bytes zstd -20, 17.468 sec., 0.119 sec., 0 bytes zstd -21, 9.179 sec., 0.014 sec., 0 bytes zstd -22, 10.418 sec., -0.163 sec., 0 bytes Minus (-) = improvement
    428 replies | 130032 view(s)
  • Jarek's Avatar
    23rd May 2020, 12:24
    Reverse engineering got to next level: https://www.extremetech.com/computing/310886-level-up-nvidias-gamegan-ai-creates-pac-man-without-an-underlying-game-engine It might also come to data compression - many simple schemes could be automatically deduced from plaintext and compressedtext ... A way to protect could be encryption e.g. this pseudorandom perturbation of tANS symbol spread.
    5 replies | 803 view(s)
  • schnaader's Avatar
    23rd May 2020, 10:44
    I voted for "private" because: - We already had so much public contests/datasets/benchmarks, something different is welcome - Interesting things can be done using dictionaries (e.g. MFilter deleting well-known ICC profiles from JPEG) - NN and ML might take the lead, but 1) it's interesting to see how they perform too and 2) I think there also will be very interesting hand-crafted entries, too
    8 replies | 544 view(s)
  • Jyrki Alakuijala's Avatar
    23rd May 2020, 10:42
    It would be a great fit for high quality photographs. Currently cameras store somewhere around 4.5 bpp, and JPEG XL can give this kind of quality in 1.5 bpp. On the other hand some cameras will want to do less computation for compression, so likely we will end up somewhere around 2 bpp. If I were a camera manufacturer I would not be leading the change. I'd wait that there is some deployment in more software-heavy technology: mobile phones, browsers, operating systems, servers, tools etc.
    14 replies | 508 view(s)
  • Shelwien's Avatar
    23rd May 2020, 02:58
    I agree that public dataset is more convenient. But it automatically requires adding decoder size to result - to counter putting part of data into decoder. However decoder size can take a considerable share of result - easily ~0.1% for zstd, which can affect the ranking. So decoder size optimization would become a part of the competition - splitting decoder and encoder, removing unnecessary compiler startup/libs, testing different compilers and their options for best size, finding the best exepacker, etc - and we're not really interested in that. On other hand, there's a practically useful option of including pre-built dictionaries in the decoder - for example, brotli and paq8px have these. These dictionaries do increase the decoder size, but also improve compression. Adding decoder size to result handicaps this, although decoder size doesn't matter for practical use - its only supposed to be a measure against exploits.
    8 replies | 544 view(s)
  • SolidComp's Avatar
    23rd May 2020, 00:13
    Thanks Jyrki. Is XL going into cameras?
    14 replies | 508 view(s)
  • Jyrki Alakuijala's Avatar
    23rd May 2020, 00:08
    Jpeg xl is favored more on butteraugli, ssimulacra and dssim, AVIF wins very clearly on psnr and vmaf. Jpeg xl tends to be more authentic, avif tends to be smoother and less artefacty. Jpeg xl is possibly better suited for portraits than avif (with current encoders). Jpeg xl's performance, utilization domain, or features cannot be estimated from experiences with jpeg xr.
    14 replies | 508 view(s)
  • Cyan's Avatar
    23rd May 2020, 00:05
    Cyan replied to a thread Zstandard in Data Compression
    That's indeed possible : zstd -r directory/ https://www.mankier.com/1/zstd#-r
    428 replies | 130032 view(s)
  • Jyrki Alakuijala's Avatar
    22nd May 2020, 23:54
    Jyrki Alakuijala replied to a thread Brotli in Data Compression
    We developed and optimized Brotli code (in 2013–2015) on gcc, so it being slow on Microsoft is not necessarily a problem in their compiler. If someone figured the reason out and made it perform better on Microsoft, we would welcome such changes in the repository. My guesswork is that there could be unfortunate inlining decisions.
    249 replies | 81707 view(s)
  • Jyrki Alakuijala's Avatar
    22nd May 2020, 23:47
    With current encoders Jpeg xl performs better at 0.4 bpp and above, avif performs better at lower bpp, likely this will even out a bit when encoders mature, but some difference like that will remain because of different filtering techniques. Jpeg xl is always at least 8x8 progressive (cannot be stored sequential in its file format), avif is always fully sequential (cannot be stored progressive in its file format).
    14 replies | 508 view(s)
  • Shelwien's Avatar
    22nd May 2020, 18:55
    Shelwien replied to a thread Brotli in Data Compression
    > The compiler that is most intriguing to me is Intel's C/C++ compiler, called Parallel Studio. > What smattering of data exist suggests that it's better than gcc, Visual Studio, clang, etc. That was true 10 years ago, but unfortunately mostly changed. Intel didn't improve their engine since 2009 or so (added GPU offload, languages extensions, new instructions, but not general-purpose optimizations - IC11 is frequently better than IC20 on scalar code). and now decided to migrate to clang completely: https://software.intel.com/en-us/articles/early-documentation-for-intel-c-compiler-based-on-the-modern-llvm-framework I'd still use it because its more user-friendly - its easier to develop with IC, then add using directives and other syntax for gcc. Also IC still sometimes wins on vector code and PGO. But in most cases, currently gcc8 wins (gcc9 seems slightly worse in my tests), on scalar code sometimes new clang wins. VS is only good for exe size optimization. 0.562s: brotli_ic19.exe -fo 1 -q 1 enwik8 0.562s: brotli_cl90.exe -fo 1 -q 1 enwik8 0.547s: brotli_gc82.exe -fo 1 -q 1 enwik8 0.921s: brotli_vc19.exe -fo 1 -q 1 enwik8 0.390s: brotli_ic19.exe -d -fo 2 1 0.437s: brotli_cl90.exe -d -fo 2 1 0.375s: brotli_gc82.exe -d -fo 2 1 0.594s: brotli_vc19.exe -d -fo 2 1
    249 replies | 81707 view(s)
  • DZgas's Avatar
    22nd May 2020, 18:40
    DZgas replied to a thread JPEG XL vs. AVIF in Data Compression
    I did not try JPEGXL but already know that AVIF is good for the Internet and Users JPEGXL is good for archiving The AVIF came out - I started testing it to compress photos, now I use JpegXR and after my tests I don’t go to AVIF 1. speed, huf this is a real death from old age if you wanted to compress a couple of hundreds of 4k photos! On my rather weak laptop it takes 5 minutes to compress a 1280x720 photo, JpegXR can in a two seconds. I know that AV1 is very slow. I did not know that it would take half an hour to compress one 4k photo from my smartphone...of course it cannot be used for real-time encoding for cameras! 2. without a doubt AVIF is better than JpegXR But I noticed that I really did not like it - the compression of dark and very little noticeable guards is just bad, But I would never have noticed it if I had not conducted tests... Even when an AVIF weighs more than a JpegXR , the JpegXR keeps more details in these unimportant places. AVIF is ideal for extra strong compression! I replaced with AVIF everything where I used WEBP. For archiving photos I use JpegXR but In the future, when "someone" supports JpegXL, I will use it. I think XnView, as they did support AVIF, easy to watch, thanks for this. Below is a photo are the same size, but AVIF want a “result without compression” in the visible areas, jpegXR spent the information on less important areas, which is much more important for me in the future. Even if later I just want to highlight the photo, strong artifacts will be in AVIF.
    14 replies | 508 view(s)
  • SolidComp's Avatar
    22nd May 2020, 15:54
    Hi all – I'm curious about any comparisons between JPEG XL and AVIF. What are the advantages and disadvantages of each? Which one is expected to be supported by web browsers? Am I correct in understanding that JEPG XL can be used as an acquisition format, in cameras? To fully replace JPEG we need an in-camera format, not just a web format like webp. Are either of these formats suitable for graphics (replacing PNG and webp), or just for photos? JPEG XL page. AVIF article.
    14 replies | 508 view(s)
  • Darek's Avatar
    22nd May 2020, 15:52
    Darek replied to a thread Paq8sk in Data Compression
    Best option for dickens file I can found for dictionary split is -e77,english.dic - this gives about 14.5 KB of gain compared to version w/o dictionary.
    82 replies | 7459 view(s)
  • Kirr's Avatar
    22nd May 2020, 15:38
    Kirr replied to a thread Brotli in Data Compression
    It's their secret undocumented setting: brotli -q 11 --large_window=30 -c >{OUT} On this page I specify each and every command line used in the benchmark: http://kirr.dyndns.org/sequence-compression-benchmark/?page=Commands
    249 replies | 81707 view(s)
  • SolidComp's Avatar
    22nd May 2020, 15:17
    SolidComp replied to a thread Brotli in Data Compression
    Kirr, what's the w30 bit at the end of the brotli versions, like brotli-11w30?
    249 replies | 81707 view(s)
  • SolidComp's Avatar
    22nd May 2020, 15:05
    SolidComp replied to a thread Brotli in Data Compression
    There's not enough careful research on these different compilers and their performance with compressors. It's a shame. The compiler that is most intriguing to me is Intel's C/C++ compiler, called Parallel Studio. What smattering of data exist suggests that it's better than gcc, Visual Studio, clang, etc. Why don't your teams at Google use it? Google can certainly afford it. It seems to be especially good at vectorization, and it has great OpenMP support to help parallelize applications. The Intel compiler is actually free for open source contributors. Linux version only. If I were better at Python or something I rig up an automated benchmarking framework.
    249 replies | 81707 view(s)
  • SolidComp's Avatar
    22nd May 2020, 14:52
    SolidComp replied to a thread Zstandard in Data Compression
    I like the new --filelist flag in Zstd 1.4.5. This makes me wonder if it's possible to give Zstd a folder and have it compress all the files in the folder individually (not compress the folder all together). Anyone know?
    428 replies | 130032 view(s)
  • SolidComp's Avatar
    22nd May 2020, 14:48
    SolidComp replied to a thread Zstandard in Data Compression
    Can someone explain the new "patch size" feature in Zstd 1.4.5? I don't understand it. Why are the percentages less than 1% in the graph? What do these values represent? (Graph screenshot below – FYI, Yann, you can save 14% on the png by losslessly optimizing it...)
    428 replies | 130032 view(s)
  • CompressMaster's Avatar
    22nd May 2020, 14:24
    Why public dataset? Well...
    8 replies | 544 view(s)
  • Kirr's Avatar
    22nd May 2020, 12:17
    Kirr replied to a thread Zstandard in Data Compression
    Thanks Yann! Congrats with releasing 1.4.5! I am already testing it, should have results in a few days.
    428 replies | 130032 view(s)
  • Cyan's Avatar
    22nd May 2020, 07:21
    Cyan replied to a thread Zstandard in Data Compression
    @Kirr made an impressive data compression comparison website
    428 replies | 130032 view(s)
  • Kirr's Avatar
    22nd May 2020, 06:56
    Kirr replied to a thread Brotli in Data Compression
    I wonder how you measure memory consumption to make this claim? In my benchmark gzip uses much less memory than brotli: http://kirr.dyndns.org/sequence-compression-benchmark/?d=all&com=yes&src=all&nt=4&bn=1&bm=tdspeed&sm=same&tn=10&bs=100&rr=gzip-9&c=brotli&c=gzip&gm=same&cyl=lin&ccw=1500&cch=500&sxm=datasize&sxl=log&sym=cmem&syl=log&button=Show+scatterplot Comparison with zopfli (and pigz -11) is irrelevant since zopfli is too far from being practical. However, I confirm that brotli usually can offer faster and stronger setting compared to any gzip level (at least on my data). E.g., comparison on human genome: http://kirr.dyndns.org/sequence-compression-benchmark/?d=Homo+sapiens+GCA_000001405.28+%283.31+GB%29&doagg=1&agg=average&com=yes&src=all&nt=4&bn=1&bm=tdspeed&sm=same&tn=10&bs=100&rr=gzip-9&c=brotli&c=gzip&gm=same&cyl=lin&ccw=1500&cch=500&sxm=ratio&sxl=lin&sym=cspeed&syl=log&button=Show+scatterplot
    249 replies | 81707 view(s)
  • Kirr's Avatar
    22nd May 2020, 06:06
    I'm thinking to test latest BCM for my benchmark (where currently v.1.30 participates). Therefore I have a question (and a feature request). Does BCM still only work with files? If so, then I'd like to request adding support for using stdio/stdout (streaming mode) for uncompressed data. In my benchmark I use all compressors only in streaming mode (reading from stdin during compression, writing to stdout during decompression). This streaming mode is important for many practical applications, where multiple tools pipe data to each other. I want my results to be relevant for such applications. Therefore, for compressors without such streaming mode, I bolt it on it via a wrapper script. A wrapper decompresses data into temporary file, then streams this file to stdout, and the TOTAL time is measured and compared with other compressors. Naturally, if a compressor can output to stdout by itself, the wrapper and temporary file is not needed and the true speed can be seen. http://kirr.dyndns.org/sequence-compression-benchmark/
    126 replies | 59999 view(s)
  • redrabbit's Avatar
    22nd May 2020, 01:42
    I tried to replace the zlib of precomp with this zlib to see how it works https://github.com/matbech/zlib Here is the benchmark https://github.com/matbech/zlib-perf/blob/master/Results.md But it failed to compile First i got this error But i i think i "solved" editing the file contrib/zlib/deflate.c and adding this line #include <stdint.h> Then i compiled again and i got this other error at 100% Any idea how to fix? Thanks
    84 replies | 13005 view(s)
  • Jyrki Alakuijala's Avatar
    22nd May 2020, 01:17
    100 mbit/s is only 10 MB/s. There you can safely use brotli 7 and get the compression done by a single core.
    428 replies | 130032 view(s)
  • Jyrki Alakuijala's Avatar
    22nd May 2020, 01:15
    I would use a simple non-context modeling compressor for usual compressed filesystems, not brotli.
    428 replies | 130032 view(s)
  • Shelwien's Avatar
    21st May 2020, 21:43
    There'd be both time limit and decompressor size limit, no GPT-2. 2nd option just doesn't let decompressor size directly affect the ranking.
    8 replies | 544 view(s)
  • algorithm's Avatar
    21st May 2020, 21:33
    Private dataset will allow large machine learning models like GPT-2. So this will be a machine learning contest. Only time will be a limiting model for very large NN models. If time is a limit maybe it will favor large statistical models instead of NN. But I am not an expert for GPT-2 like stuff. Other people should know better about it.
    8 replies | 544 view(s)
  • Alexander Rhatushnyak's Avatar
    21st May 2020, 21:30
    The question is not only about the English Texts set. There will be a few other test sets. Option 1: -- Test data are 100% available, participants download the test data just once, and don't have to worry about building a testing set. -- It's up to participants to decide what data, if any, to embed into their executables. -- Compressed size of decompressor is added to the compressed size of the test set. Option 2: -- Test data used for ranking are unavailable to participants, they have to build their testing sets themselves (organizers will provide some samples). -- There's a limit on decompressor size (most likely ~16 MB) and if your decompressor is smaller, it's your problem. -- Encourages pushing dozens of megabytes of data into executables.
    8 replies | 544 view(s)
  • Shelwien's Avatar
    21st May 2020, 19:48
    English plaintext compression task. 1. Public dataset: - we have to add decompressor size to compressed size; - encourages compiler tweaks, discourages speed optimization via inlining/unrolling (lzma.exe.7z: 51,165; zstd.exe.7z: 383,490) - encourages overtuning (participants can tune their entries to specific data); - embedded dictionaries are essentially blocked; 2. Private dataset: - decompressor size has to be limited for technical reasons (<16MB?) - embedded dictionaries can be used fairly - embedded dictionaries actually improve compression results (see brotli vs zstd benchmarks) - we can post hashes of private files in advance, then post the files after end of contest Its for an actual contest that is being prepared. Please vote.
    8 replies | 544 view(s)
  • serenalai2005's Avatar
    21st May 2020, 18:14
    What did the dinosaur say while it was being compressed? RAR. Source: https://www.reddit.com/r/Jokes/comments/7s8ojr/i_bought_winrar/
    6 replies | 1562 view(s)
  • suryakandau@yahoo.co.id's Avatar
    21st May 2020, 17:33
    everything is ok
    82 replies | 7459 view(s)
  • Darek's Avatar
    21st May 2020, 17:20
    Darek replied to a thread Paq8sk in Data Compression
    It looks suspicious - are you shure that the program finish OK? There no any crash at the end?
    82 replies | 7459 view(s)
  • Jyrki Alakuijala's Avatar
    21st May 2020, 16:13
    In JPEG XL we decided to take as much out of the SIMD as we can and create progressive features that are more advanced than those of original JPEG. These decisions definitely do not play an advantage in simplicity and have given headaches with exposing SIMD-processing bugs in old compilers. For one clang, a two year old version (clang 6) doesn't work well for everything we want to do, but a 1.5 year old version (clang 7) already works. In JPEG XL simplicity comes in different forms than code: no specialized 8-bit mode, no yuv420 mode, each channel modeled as a float (instead of 8 or 10 bit integer), a psychovisual guarantee for loss to start happen everywhere in the image at the same time -- where some codecs start to destroy dark green and red/magenta areas first, one codec destroy brown areas first and then others, etc. Simplicity of use vs. simplicity of compiling. Simplicity of performance evaluations, too: other codecs publish performance eval in yuv420 mode with 8-bit dynamics, where we show them constantly at HDR/wide-gamut-ready yuv444/32 bit float dynamics. We tried to hide some of the increase in binary footprint, like JPEG XL uses brotli for metadata compression, which gives rather nice compression gain using technology that already sits in the browser, and we build the reference encoder using the same color management library that skia uses.
    152 replies | 35452 view(s)
  • vteromero's Avatar
    21st May 2020, 13:05
    vteromero replied to a thread VTEnc in Data Compression
    VTEnc v0.2.0 has been released! (CHANGELOG) The most remarkable feature of this version is the inclusion of encoding parameters, which need to be provided when using encoding and decoding functions. Encoding parameters give users the ability to specify the way sequences are encoded, resulting in different encoding/decoding speeds and encoding ratios. For now, there are 3 encoding parameters: allow_repeated_values, skip_full_subtrees and min_cluster_length. I'm writing a blog post to explain them in depth, I'll post the link in this thread once it's published. Here are the benchmark results on gov2.sorted data set: (*) VTEnc's results on "Encoding speed vs ratio" chart are for the following values of the encoding parameter min_cluster_length: 1, 2, 4, 8, 16, 32, 64, 128 and 256. (**) VTEnc's decoding speed is for min_cluster_length = 256.
    21 replies | 4411 view(s)
More Activity