What are the lightest (CPU and memory) general purpose compressors?

[SolidComp]

Hi all,

The usual benchmarks report compression ratio and speed (and maybe decomp speed). I'm interested in the lightest, most efficient general-purpose codecs in terms of CPU and memory usage. I can't find any valid benchmarks for this – do you know of any?

Matt Mahoney's large text compression benchmark isn't rigorous enough to determine the most efficient codecs. The test hardware isn't held constant – one codec might be tested on a completely different CPU and OS than another codec, and with a different amount of RAM. This makes the results completely invalid and useless. And the benchmark doesn't report CPU usage. Moreover, many of the codecs are extremely old releases – e.g. the "gzip" is in the LTCB is an ancient 2007 version of gzip for Windows: http://gnuwin32.sourceforge.net/packages/gzip.htm

The gzip we need would be the latest version of zlib, which is what most web servers use. Which reminds me that one way of getting at the most efficient compressor would be to match zlib's CPU/memory efficiency while beating its compression ratio, (since zlib/gzip are about the lightest codecs we have right now in terms of memory and CPU). Or to use even less CPU/memory than zlib at the cost of worse compression ratios. Willy's SLZ seems to do this: https://encode.su/threads/2575-SLZ-s...ble-compressor

Are there are any codecs that use less CPU and RAM than zlib but achieve better compression ratios? Even more awesome would be that it also be faster than zlib. (SLZ is much faster but doesn't compress as well.)

Thoughts? Codecs? For web servers it would be nice to be able to efficiently compress dynamic content on the fly, and Zstd ad brotli don't seem to suitable for that.

Cheers,

Joe

[cfeck]

I can't find any valid benchmarks for this

You can benchmark many compressors with https://github.com/powturbo/TurboBench, https://quixdb.github.io/squash-benchmark/, https://github.com/ahti/fsbench/tree/master/src/codecs, https://github.com/inikep/lzbench.

Are there are any codecs that use less CPU and RAM than zlib but achieve better compression ratios? Even more awesome would be that it also be faster than zlib.

You aren't actually new to this forum, so I am a bit irritated why you still expect a Holy Grail of compression algorithms that weren't already discussed. EDIT: added more benchmark tools

[Gonzalo]

Are there are any codecs that use less CPU and RAM than zlib but achieve better compression ratios? Even more awesome would be that it also be faster than zlib. (SLZ is much faster but doesn't compress as well.)

Thoughts? Codecs? For web servers it would be nice to be able to efficiently compress dynamic content on the fly, and Zstd ad brotli don't seem to suitable for that.

Cheers,

Joe

Well, AFAIK, that is the exact use case for which ZSTD and brotli were designed. One is backed by Facebook and the other by Google so I don't think you're gone get much more than that...

Good luck in any case!

[SolidComp]

Well, AFAIK, that is the exact use case for which ZSTD and brotli were designed. One is backed by Facebook and the other by Google so I don't think you're gone get much more than that...

Good luck in any case!

No brah. Zstd and brotli use way more CPU and memory than gzip, so they're not the sort of codecs I'm pining for. They're good with their compression ratios and speed at some settings. But we can't use either of them for streaming compression of dynamic web content – they'd use way too much CPU and memory (some websites use brotli to precompress some "static" css and js files). Even gzip (zlib, technically) was using too much CPU for Willy's tastes re: HAProxy, so he developed SLZ, which creates gzip files faster while using less resources.

[Jyrki Alakuijala]

brotli:0 and zstd:1 encode about 3x faster than gzip:1

brotli:0 and zstd:1 compress more than gzip:1

brotli and zstd both decode faster than gzip -- (((brotli benefits from superscalar cpus, and zstd from 64-bit cpus)))

Facebook, Google Search and LinkedIn (and others) use brotli for streamed dynamic compression -- usually at quality 5 or so and with window of about 512 kB for dynamic compression, so they have plenty of room to go faster and with less memory when there is a need.

Apple, Microsoft, Google and Mozilla (and others) can decode the dynamic brotli streams.

Perhaps we can retire gzip after all.

[SolidComp]

brotli:0 and zstd:1 encode about 3x faster than gzip:1

brotli:0 and zstd:1 compress more than gzip:1

brotli and zstd both decode faster than gzip -- (((brotli benefits from superscalar cpus, and zstd from 64-bit cpus)))

Facebook, Google Search and LinkedIn (and others) use brotli for streamed dynamic compression -- usually at quality 5 or so and with window of about 512 kB for dynamic compression, so they have plenty of room to go faster and with less memory when there is a need.

Apple, Microsoft, Google and Mozilla (and others) can decode the dynamic brotli streams.

Perhaps we can retire gzip after all.

Lots of issues here...

1. I keep talking about CPU and memory use, and you keep not mentioning either of those things. You keep talking about speed on encode or decode, or ratio. That's fine. Those are major factors in how codecs are evaluated. I'm saying that I also care about efficiency, defined as CPU and memory use.

2. Silicon forests like Facebook, Google, and LinkedIn are not representative users. Let's take Facebook as an example. Facebook is a testament to the power of money to compensate for bad engineering. More specifically, the power of money to secure enough good engineering to compensate for bad engineering. Facebook takes millions of dollars every year and lights it on fire. Maybe hundreds of millions. Their particular method of lighting money on fire is somewhat common in the industry, but with the addition of PHP. They spend a lot more on servers than they ought to because they use absurd interpreted or JITed programming languages like PHP, Hack, the bytecode interpreter they built, etc.

The spend a lot more on bandwidth than they should because they shove massive amounts of unused crap down the pipe to the user. Just go to Facebook right now in Chrome, and use the new Coverage tool/tab. Look at all the unused CSS and JS – all of it brotlified, by the way, which is unbelievably ironic. You should find that most of the CSS and JS is unused, and that there's more than 1 MiB of it. (The Chrome Developers page I linked to for the Coverage tool doesn't actually explain how to enable it / where to find it. In Dev Tools, click on the three-dot menu in the upper right (they're vertical dots). Then hover over More Tools, which expands into another menu and choose Coverage. It will then show it at the bottom with the Console.)

3. According to Cloudflare's initial tests, zlib 6 (the default) is 2-3 times as fast as brotli 5. Влад said that for large files, brotli 5 could make sense in terms of the trade-off between better compression ratio and slower speed. But he didn't comment on CPU and memory use, which is, again, my focus in this thread.

Companies waste enormous amounts of money on servers and bandwidth because early-21st century web developers are terrible at web development. Site owners don't know how badly their sites are built and how much faster and more reliable they would be if "front-end engineering" were an actual thing. For historical reasons that are unclear, web developers locked onto the antipattern of having separate CSS and JS files, then having maybe twelve of each, and filling them with a bunch of stuff that would never be used or executed. The incantation "caching" seems to have distracted generations of web developers into not collecting any data to find out if this approach was justified. The web is a fractal of waste and hipster "engineering". "Servers are cheap" is actually their slogan, the reason they give for using Ruby and PHP and not hiring .NET developers. (This link is a better example than the Expensify post.)

So, brotli... Yeah, it's better at compressing than gzip. I certainly couldn't have built it – I'm a social scientist, and I admire anyone who can do things I can't, like all the things you've built. The issue here is that brotli gets us a few percentage points over gzip in a context where the browser is having to decompress and parse five times the amount of content it actually needs (the numbers can be much worse than that). So brotli won't make much of a difference if we can't get web developers to learn math. (And like I've said before, brotli needs a new dictionary to achieve its true potential. I volunteer to help.) Brotli can save us 10 or 20 KiB, but rational AMP-y web development can save us hundreds of KiB. (And even with AMP you'll see tons of unused CSS in the head.) I mean all a CMS or whatever has to do is match CSS selectors to the HTML in the same file, and yet they don't bother. Someone should build a CMS that compiles optimized single-file web pages and apps – it would blow everything else we have away. WordPress would look like an oxcart.)

Joe Duarte

[dnd]

You have all the results served on a plate: "Static/Dynamic web content compression benchmark".
It is already known to you.
This is a pure practical benchmark without any other influencing factors like network latencies, multitasking, web server overhead, minifying,...

The only difference, it is not coming from companies like Cloudflare, but you can verify the results yourself on your platform.

> According to Cloudflare's initial tests, zlib 6 (the default) is 2-3 times as fast as brotli 5.
This is simply misleading, because "brotli 4" is already compressing better than "zlib 6" and is also faster.

You can redo the benchmark yourself with "Turbobench Compression benchmark" which can also reports the memory usage.
There is little sense to consider libraries (ex. zstd) not supported by the major web browsers.
Brotli dictionary is static and can be shared by all threads.

If you are not facebook,linked-in,... it is cheaper to buy more cpu/ram than to switch from zlib to brotli.
As I've demonstrated in another thread, brotli is bringing only little if anything at all to web browser users.

[Jyrki Alakuijala]

Static/Dynamic web content compression benchmark

This matches with my own experiments, and I believe the above benchmarking format is the most important single compression experiment. It is also the use case that brotli was designed for.

(((Repeating this kind of experiment with Asiatic languages might give a yet another picture into compression.)))

> According to Cloudflare's initial tests, zlib 6 (the default) is 2-3 times as fast as brotli 5.

This is simply misleading, because "brotli 4" is already compressing better than "zlib 6" and is also faster.

Does anyone have an insight why they chose a commutative scoring function? To me it seems that it is not possible to draw A-vs-B conclusions from a commutative scoring function.

[terrelln]

1. I keep talking about CPU and memory use, and you keep not mentioning either of those things. You keep talking about speed on encode or decode, or ratio. That's fine. Those are major factors in how codecs are evaluated. I'm saying that I also care about efficiency, defined as CPU and memory use.

Single threaded Zstd and Brotli at the lowest level are faster to both compress and decompress, which means they use less CPU than zlib/gzip. However, they both use more memory, since they have a larger window size.

If you are looking for an algorithm that is both faster than zlib, and uses less memory, check out lz4.

[Jyrki Alakuijala]

Single threaded Zstd and Brotli at the lowest level are faster to both compress and decompress, which means they use less CPU than zlib/gzip. However, they both use more memory, since they have a larger window size.

If you are looking for an algorithm that is both faster than zlib, and uses less memory, check out lz4.

You can reduce the window size down to 1024 bytes with stock brotli encoder if memory is tight (or if data has not repetitions).

You can do this with '--lgwin 10' or programmatically through https://github.com/google/brotli/blo...c/params.h#L26

Going very low on the window size is harmful for total system performance and because of this we have not optimized that use case. Going lower than ~64 kB is very likely to increase the total use of system resources including memory and memory bandwidth use, and gzip's behavior of going lower than 32 kB comes from the times when our primitive cpus didn't have L1 caches of 32 kB and no 256 kB L2 caches. As an example of this, deflate64 compresses and decompresses faster than deflate, and compresses more -- the only difference is upping the window from 32 kB to 64 kB there.

Going lower than 4 kB or so will cause the compression to degrade more than the few bytes saved -- you will use more memory to represent the compressed symbols. The same with LZ4, when you actually compress you are using more bytes for the compressed results, and thus the overall memory use and memory bandwidth use will be larger.

[SolidComp]

You have all the results served on a plate: "Static/Dynamic web content compression benchmark".
It is already known to you.

Dude. If it was already known to me, I wouldn't have asked. Now, I do realize after seeing this webpage that I've seen it before. I simply forgot about it – sorry.

However, I don't think I've had anything served to me on a plate, because the results on that page are obsolete now, with old versions of brotli, zstd, etc. Also, there's no CPU usage reported, and that's what I keep talking about.

> According to Cloudflare's initial tests, zlib 6 (the default) is 2-3 times as fast as brotli 5.
This is simply misleading, because "brotli 4" is already compressing better than "zlib 6" and is also faster.

What do you mean? Do you mean brotli 4 like right now, today, a newer version of brotli than tested by Cloudflare? That may be true – I haven't kept up. Brotli has had very few releases compared to zstd, and they don't provide binaries, but rather make people compile it using all kinds of weird tools. I cited brotli 5 because Jyrki cited brotli 5. I was just going with his baseline. In any case, yeah, if brotli is faster now, the Cloudflare tests are no longer valid for the same reasons I complained about re: your benchmarks.

[SolidComp]

Single threaded Zstd and Brotli at the lowest level are faster to both compress and decompress, which means they use less CPU than zlib/gzip...

How is that? You mean because they finish sooner? But "sooner" and "faster" describe a wide range of possible values, and when they are compressing, how much CPU they use could matter a great deal, and how much it matters will depend on the actual values reality forces us to substitute for "sooner" and "faster".

[SolidComp]

You aren't actually new to this forum, so I am a bit irritated why you still expect a Holy Grail of compression algorithms that weren't already discussed. EDIT: added more benchmark tools

Man, people keep thinking I know all kinds of shit I don't know. Damn it Jim, I'm a doctor not a compression guru.

What I do know is that there's a ton of research out there that is never discussed on this forum. So I'd never assume that I know about everything going on in compression research just from reading this forum. I know that there are real geniuses here and some big names, but I'm telling you I see lots of papers on ResearchGate alone, and I think a lot of those researchers are not here. One reason might be that a lot of the researchers seem to be Asian, in China or Japan mostly, and maybe they don't feel comfortable with English-language forums, or maybe they don't even know about encode.su. The world's a big place. (And as far as their papers being in English, I think sometimes people pay for translations, and sometimes obviously their English is pretty bad.)

If you search on ResearchGate for lightweight compression or GPU compression, you get tons of papers. (I assume Google Scholar would work just as well, but I've been impressed with ResearchGate lately, especially the nicer formatting and readability.)

Here's an interesting one: https://www.researchgate.net/publica...Implementation

And another one: https://www.researchgate.net/publica...or_GPU_revised

These are cool ideas. The second one is Eastern European, so maybe it's yours or someone else's on the forum – I don't know who everyone is here.

[dnd]

However, I don't think I've had anything served to me on a plate, because the results on that page are obsolete now, with old versions of brotli, zstd, etc. Also, there's no CPU usage reported, and that's what I keep talking about.

There are no major changes for brotli,zstd since the last benchmark. I've redone the skylake benchmark today and there are no changes in compression ratio or speed.
Interesting is the now added "brotli,5".

This is a single thread in memory benchmark (no IO or sys calls), so the cpu usage is corresponding to the timings of the benchmark.

Again compressors other than gzip,deflate,br compatible, like zstd,lz4,... are only interesting as indication.
You can't use them in a web server.

To my knowledge the "Static/Dynamic web content compression benchmark" is the only practical benchmark comparing different web compression libraries with memory usage reporting.
The cloudflare benchmark is focusing on the not compression interesting small files.
There is only a short info about the benchmark.