Squash and the Squash Benchmark

[nemequ]

A while back I started working on a project called Squash, and today I'm pleased to announce the first release, version 0.5. Squash is an abstraction layer for general-purpose data compression (zlib, LZMA, LZ4, etc.). It is based on dynamically loaded plugins, of which there already a pretty decent number:

• Brotli
• libbsc
• bzip2
• CRUSH
• DENSITY
• Doboz
• FastARI
• FastLZ
• Gipfeli
• LZ4
• LZF
• liblzg
• LZHAM
• LZJB
• liblzma
• LZO
• ms-compress
• pithy
• QuickLZ
• Snappy
• wfLZ
• zlib
• zling
• ZPAQ
• Zstandard

Additionally, a new and greatly improved version of the Squash Benchmark is online. The benchmark includes every compression level for every codec Squash has (except the ones from disabled plugins) on 8 different machines. That's about 40,000 different data points, and I'm planning on adding more machines soon—there will probably be about 14 within the next couple weeks.

The API isn't final yet, but I don't think it will change much—I plan to make some additions, but not changes. I'm rolling out a release now in the hope that it encourages people to give it a try, since I don't want to commit to API stability until a few people have given it a try.

So, why should you care? Well:

• It makes it much easier for people to try more exotic codecs, since they don't need to write new code for each one.
• Squash provides most of the fluff that isn't compression or decompression code, like a buffer to buffer API for codecs with streaming APIs, a command-line interface, I/O handling (including memory-mapped files where appropriate), etc. Very happy to take feature requests here.
• Free bindings to any language Squash supports. Currently this is only C and Vala, but I'm really hoping people come along to write bindings for their favorite language. IMHO bindings are best written by people who really know the target language and its idioms, so this isn't really something I can do on my own for too many languages.
• Inclusion in the Squash Benchmark.
• Pretty good testing of your code. Between the unit tests in Squash and the benchmark I've found a decent number of bugs in several libraries.

I'm very much open to suggestions, questions, flames, etc. I'd prefer you file feature requests or bugs in Squash's issue tracker but here will work, too.

[Gonzalo]

Some of those I didin't even know

Very good job!

Could you include the binaries you are using? For windows32...

Some time ago you talked about include squash as a plug-in for 7z... Have you plans to do it some time nearby?

[nemequ]

Could you include the binaries you are using? For windows32...

Sorry, there is currently no support for Windows. It actually wouldn't be very difficult (mainly just creating a Windows backend for the memory mapped files API), and I had a Windows version building a while back (it's actually why I switched from autotools to CMake), but I'm not really sure where plugins should go, how it should be packaged/installed (nuget maybe?), how to deal with CI (appveyor?), etc.

If anyone is willing to help with a Windows port it would be greatly appreciated. That and creating additional language bindings are the two big areas where I could really use help, though of course I'm happy to accept all contributions.

Some time ago you talked about include squash as a plug-in for 7z... Have you plans to do it some time nearby?

Nope. I was thinking about it as a stress test, but between the unit tests and the benchmark I think the core is pretty solid, and I don't actually use 7-zip so it's hard for me to care. There is a command line utility (`squash`) which is a bit like gzip/bzip2/xz/etc.

[m^2]

Does gipfeli work for you?
For me, sometimes yes, usually no and I wonder if I use it right.

[nemequ]

Does gipfeli work for you?
For me, sometimes yes, usually no and I wonder if I use it right.

Yes, the only plugins which don't work reliably are libzling and wfLZ, which is why they are disabled by default until the libraries are fixed.

Gipfeli didn't use to work, but a it has been fixed for a while. See https://github.com/google/gipfeli/pull/1.

[load]

gipfeli? brotli? zopfli? i wonder why they've chosen such stupid names. what's next? semmeli?

nice work, thanks.

[Nania Francesco]

If possible to test LZA archiver 0.82 ?

[inikep]

If possible to test LZA archiver 0.82 ?

This is a benchmark of compression algorithms supported by Squash. It will be possible to test LZA only if you will release sources of LZA. Actually I have never used your compressor because it is closed source. The closed source compressors will never be popular e.g. lzturbo seems to be better than LZMA but it's not widely used. In the other hand LZ4 is used in many projects after its sources were opened.

[nemequ]

If possible to test LZA archiver 0.82 ?

Unless I'm mistaken, LZA Archiver is a proprietary Windows-only executable. Given that, it's unlikely.

In order to be included the benchmark the software must be supported by Squash. It is certainly possible to write a plugin for proprietary software—I'm not willing to do it myself, but I might be willing to consider a pull request. However, it's not actually necessary for plugins to be distributed with Squash. As long as you provide a plugin (shared library with a few callbacks for Squash) and an ini file in the location Squash looks it should work regardless of whether the plugin was distributed with Squash. That is something I've always intended to support, and as soon as Squash hits 1.0 the API/ABI for plugins will be stable.

Being an executable instead of a library is a problem. Technically you could write a plugin which did the whole fork()/exec() thing, but honestly that's an ugly hack and you would see a pretty big performance hit. For smaller files, times would be dominated by the overhead, but even for larger files you'll would have to do a lot of extra copying.

Finally, being Windows-only is a problem. As discussed earlier, Squash currently doesn't work on Windows (though it shouldn't be too difficult to change that). The benchmark, OTOH, is run exclusively on Linux, with various distros and architectures. Honestly, this is something open-source makes much easeir... for proprietary software you would currently have to provide x86_64 and armv7l versions, and I intend to add x86 and armv6l (and maybe mips) machines soon. All for Linux, of course.

If you're still interested, you can find lots of examples in the plugins/ directory of Squash, and I'm happy to answer questions.

Note that there is also an entry in the FAQ for this question, though it's not quite as detailed as what I wrote above.

[hexagone]

Nice work.
Suggestion: the "Compression ratios vs. ..." graphs could use a log scale for clarity.

[nemequ]

Nice work.
Suggestion: the "Compression ratios vs. ..." graphs could use a log scale for clarity.

Thanks for the feedback. You can already toggle between linear and logarithmic scales by clicking on either axis' label. I know this isn't obvious, or really even discoverable (unless you read the FAQ)—if you can think of a better way to expose it I'd be happy to listen (or even happier to accept a patch, of course).

I don't want to switch the default because I think that linear is probably better for most people. Logarithmic tends to be better if all you care about is compression ratio, not speed. Linear seems better for comparing fast codecs. Since linear scales are more natural for people I think it is best to stick to that.

[Matt Mahoney]

I think linear is best for compression ratio and logarithmic is best for time and memory (where the range is much wider) like the graphs in http://mattmahoney.net/dc/text.html

[nemequ]

I think linear is best for compression ratio and logarithmic is best for time and memory (where the range is much wider) like the graphs in http://mattmahoney.net/dc/text.html

Linear is probably always best for ratio, but IMHO linear is also better for speed (and memory, though I don't have that yet) if you're mainly interested in speed not ratio—i.e., if you're looking at LZ4 not ZPAQ. Logarithmic also has the disadvantage that it isn't natural for people unless they are very used to the concept… I don't think it's hard for people to understand, just that there is a cognitive barrier there which makes it a bit tricky to really internalize the information accurately.

[nemequ]

3 quick pieces of news:

1. The benchmark already has almost 10,000 unique visitors, mostly thanks to spending most of yesterday at the top spot for reddit's programming subreddit (the thread currently has a score of just over 400). That's a pretty good amount of exposure for libraries with included plugins. If you want even more, please consider upvoting that and/or this submission to Slashdot.
2. I added a round-trip time (which factors in both compression and decompression speed) vs. compression ratio chart.
3. I just merged a plugin for yalz77, which I hadn't heard of. Benchmark should be coming within the next few days.

[tkatchev]

yalz77 is a codec I wrote when I couldn't find an existing library to fit my requirements:

a) super-simple, only about 300 lines of portable C++ header-only code. (No assumptions about the platform or data.)
b) "idiot-proof" API -- it manages buffers and figures out message boundaries automatically.

Surprisingly, its performance is actually not that bad, and sometimes even competitive with "real" codecs. (Well, at least on the limited number of data sets and codecs I've tested it with.)

[m^2]

Yes, the only plugins which don't work reliably are libzling and wfLZ, which is why they are disabled by default until the libraries are fixed.

Gipfeli didn't use to work, but a it has been fixed for a while. See https://github.com/google/gipfeli/pull/1.

One more question, have you checked if it decompresses correctly? I'm getting CRC mismatches all the time. I looked at your code and I do the same things.
Does squash benchmark have some way to verify if decompression really succeeded that I can try on my PC?

[nemequ]

One more question, have you checked if it decompresses correctly? I'm getting CRC mismatches all the time. I looked at your code and I do the same things.
Does squash benchmark have some way to verify if decompression really succeeded that I can try on my PC?

I have several unit tests in Squash which use memcmp to check the decompressed output against the input (which all the plugins, including gipfeli, pass). The benchmark only checks that the size matches. You could use the squash(1) command line tool to manually compress/decompress a file using gipfeli and verify the sha1sum:

$ squash -kc gipfeli  alice29.txt alice29.txt.gipfeli && squash -kdc gipfeli  alice29.txt.gipfeli alice29.txt.ungipfeli && sha1sum alice29.txt  alice29.txt.ungipfeli
37a087d23c8709e97aa45ece662faf3d07006a58  alice29.txt
37a087d23c8709e97aa45ece662faf3d07006a58  alice29.txt.ungipfeli

[m^2]

Not easy to build...
My file manager highlighted an executable: autoconf. Doesn't work requires bash. Then I spotted cmake, that's better.
Some program dependencies have to be installed. Plugins have to be downloaded as well as a specific library. Recursion makes things harder as errors don't tell which subproject has a missing dependency. There are no docs. I ended up disabling all plugins but one, that part was easy.
And then compilation failed on my PC:

m% make
[ 2%] Building C object squash/CMakeFiles/squash0.5.dir/buffer.c.o
/usr/home/m/Downloads/squash-master/squash/buffer.c:122:11: error: no member named 'fd' in 'struct _SquashBuffer'
buffer->fd = -1;
~~~~~~ ^
1 error generated.
*** Error code 1

Stop.
make[2]: stopped in /usr/home/m/Downloads/squash-master
*** Error code 1

Stop.
make[1]: stopped in /usr/home/m/Downloads/squash-master
*** Error code 1

Stop.
make: stopped in /usr/home/m/Downloads/squash-master

Oh well, fd is not there, no preprocessor macro can change it and I got a bit lost.

[nemequ]

Not easy to build...
My file manager highlighted an executable: autoconf. Doesn't work requires bash. Then I spotted cmake, that's better.

Actually, no autoconf. I guess you meant configure, which is just a wrapper script I wrote to make CMake look more like autotools because CMake's interface sucks. TBH I thought bash was at least available, though not the default, on BSD (based on the fact that you have mmap but not mremap, I assume you're on a non-Linux UNIX-like OS, OS X has bash, so odds are a BSD). I'll try to take a look at making csh work at some point—it should be possible, it's just escaping arguments becomes a bit of a nightmare.

Some program dependencies have to be installed. Plugins have to be downloaded as well as a specific library. Recursion makes things harder as errors don't tell which subproject has a missing dependency.

If you used the release (instead of git) most of the plugins are already there. If not, `git submodule update --init --recursive` (which is what autogen.sh does) will pull in everything. I'm in the process of adding submodules for all the libraries as a fallback for when system libraries aren't available, I believe only liblzma (from xz-utils) is missing, so current git (or the next release) should be much easier.

There are no docs.

For building, yes, you're right. I'll start writing some. TBH I just assumed people who aren't on Windows would just see autogen.sh/configure, run the autogen.sh just like they do for autotools projects, and be done with it. Meanwhile, if you have bash: `./autogen.sh && make` for git or `./configure && make` for the release. If you don't have bash, `git submodule update --init --recursive && cmake . && make`. Disabling the lzma plugin for now should work, or install liblzma.

And then compilation failed on my PC:

Oh well, fd is not there, no preprocessor macro can change it and I got a bit lost.

It looks like a bug in the fallback code when a system has mmap but not mremap. Just fixed in git.

[nemequ]

There are no docs.

There are now: Building Squash.

Edit: and I just added a copy of liblzma as a submodule, there should now be no external hard dependencies—just recursively init the submodules and you should be good.

[m^2]

Actually, no autoconf. I guess you meant configure, which is just a wrapper script I wrote to make CMake look more like autotools because CMake's interface sucks. TBH I thought bash was at least available, though not the default, on BSD (based on the fact that you have mmap but not mremap, I assume you're on a non-Linux UNIX-like OS, OS X has bash, so odds are a BSD). I'll try to take a look at making csh work at some point—it should be possible, it's just escaping arguments becomes a bit of a nightmare.

FreeBSD. Bash is available in ports, it's just that I'm not fond of having to install it.

If you used the release (instead of git) most of the plugins are already there. If not, `git submodule update --init --recursive` (which is what autogen.sh does) will pull in everything. I'm in the process of adding submodules for all the libraries as a fallback for when system libraries aren't available, I believe only liblzma (from xz-utils) is missing, so current git (or the next release) should be much easier.



For building, yes, you're right. I'll start writing some. TBH I just assumed people who aren't on Windows would just see autogen.sh/configure, run the autogen.sh just like they do for autotools projects, and be done with it. Meanwhile, if you have bash: `./autogen.sh && make` for git or `./configure && make` for the release. If you don't have bash, `git submodule update --init --recursive && cmake . && make`. Disabling the lzma plugin for now should work, or install liblzma.



It looks like a bug in the fallback code when a system has mmap but not mremap. Just fixed in git.

I used the most direct download - the zipped repo from the front page. It's good that releases contain all dependencies.

I'd like to try trying the fix out, but I'm too tired today. Got overworked again...

[nemequ]

FreeBSD. Bash is available in ports, it's just that I'm not fond of having to install it.

Well, I filed an issue—it's low priority, but I'll probably get to it eventually. Meanwhile, you'll just have to like with the insanity that is cmake's CLI.

I used the most direct download - the zipped repo from the front page. It's good that releases contain all dependencies.

Sigh, stupid GitHub. AFAICT there is no way to disable that button, or make it point to a release, or make it include the git data (not even a shallow clone) so you can init the submodules yourself. I've complained to GitHub, but that's about all I can do

Anyways, for now I suggest just using git and doing a `git submodule update --init --recursive`. I'll put together a 0.5.1 release soon.

[Gonzalo]

Can somebody provide bins? Linux-x86. Having a hard time with compiling...
Thanks in advance!

[nemequ]

Can somebody provide bins? Linux-x86. Having a hard time with compiling...
Thanks in advance!

I don't really have a good way to provide x86 binaries. Compilation should be easier now if you use git instead of the 0.5.0 release since there are no hard external dependencies, but I just uploaded a pre-release tarball of current git master (which will be 0.5.1 very soon, just need to fix one last issue). For directions see https://github.com/quixdb/squash/blo...cs/building.md.

That said, I would really like to hear about what problems you're having so I can try to fix any issues. If you don't want to post them here feel free PM me here, e-mail me (my username for this forum at gmail.com will work), drop by #squash on freenode, or file an issue.

[Gonzalo]

I don't really have a good way to provide x86 binaries. Compilation should be easier now if you use git instead of the 0.5.0 release since there are no hard external dependencies, but I just uploaded a pre-release tarball of current git master (which will be 0.5.1 very soon, just need to fix one last issue). For directions see https://github.com/quixdb/squash/blo...cs/building.md.

That said, I would really like to hear about what problems you're having so I can try to fix any issues. If you don't want to post them here feel free PM me here, e-mail me (my username for this forum at gmail.com will work), drop by #squash on freenode, or file an issue.

Thanks. The trouble is basically that I don't know what to pass to the first script. I didn't used cmake at all and I didn't found nothing clear in the web. So the source cannot be configured nor compiled...
I know this is embarrassing
Do you know just yesterday I found i can use /dev/* for testing? Now it rocks

[nemequ]

Thanks. The trouble is basically that I don't know what to pass to the first script. I didn't used cmake at all and I didn't found nothing clear in the web. So the source cannot be configured nor compiled...

Ah. You don't usually *have* to pass anything. By default it will install to /usr/local, which is a pretty sensible thing to do on most distros. You can pass --help to see a list of arguments and an explanation of what they do, though it might be a bit hard to really understand for a beginner.

I didn't used cmake at all and I didn't found nothing clear in the web. So the source cannot be configured nor compiled...

There isn't a lot of documentation out there about CMake, and from a user's perspective CMake really sucks. The configure script in Squash is basically an autotools-style wrapper around CMake, so if you were searching for CMake documentation that explains why you wound have trouble. If you want to use the configure script (which I recommend) you should pretty much think of Squash as autotools-based… the configure script accepts most of the same arguments which are standard for autotools.

I know this is embarrassing

Not at all. It sounds like you're still learning this stuff, that's definitely nothing to be embarrassed about. TBH it's so natural for me (I've been using Linux for ~ 15 years) that it can be hard to write good documentation because I just don't think to mention a lot of things. I'll try to be a bit more helpful in the "Building Squash" document, or at least link to somewhere which explains it better. Thanks for calling my attention to the shortcoming.

Do you know just yesterday I found i can use /dev/* for testing? Now it rocks

I wouldn't do that if I were you. /dev is for device files (like the file representing your hard drive, which is probanly /dev/hda or /dev/sda), you should probably stay out of there. /usr/local is a good prefix to use.

[nemequ]

I'm happy to announce I have just released Squash 0.6.0!

Highlights of this release include:

• Four new codecs: yalz77, csc, ncompress, and copy (which just does a memcpy instead of compressing).
• Created in-tree copies of all modules, which simplifies compilation.
• Use conformant array parameters for buffers (see http://tinyurl.com/p4lr4ze if you're not familiar with the concept).
• Added a new API for option parsing and introspection which makes writing plugins easier.
• Updated lots of plugins to the latest stable versions, including big improvements for LZ4, DENSITY, Zstandard, and Brotli.

The Squash Benchmark has also been updated. Highlights there include:

• More data—we're now up to 52,164 different configurations (28 datasets, 41 codecs at 207 different levels on 9 machines)!
• An awesome new "Transfer + Processing" chart which is great for people who just want to use compression to read or write their data faster, and don't necessarily care about disk space or bandwidth.
• A new "Optimial Codec" chart showing the fastest codec at different compression ratios.
• Codec details (most notably library version/revision numbers) are now presented.

This release took a lot longer than I expected, sorry about that. It has been basically ready to go for a couple months, but work got *really* busy and I couldn't find time to update the benchmark; things have calmed down a bit (for now), so hopefully the next release will be a bit quicker.

You can see the plans for future versions of Squash in the issue tracker on GitHub. Milestones are currently set up for 0.7 and 1.0, though things may of course be added or removed before those releases. If you have other ideas please file issues (or, even better, pull requests!) for them, or let me know here and I'll file them.

For those of you who would like something to re-tweet, the Twitter announcement is at https://twitter.com/nemequ/status/634967341189599232, or https://twitter.com/nemequ/status/634967589031022592 if you're more interested in the benchmark than the library.

I've also added an announcement-only mailing list for anyone who would like to be notified of new versions via e-mail.

[Jyrki Alakuijala]

> An awesome new "Transfer + Processing" chart

Everything in the squash benchmark is awesome. Thanks for doing this!

A small improvement proposal for the transfer + processing chart:

Browsers (and many other transfer systems) start decompressing and processing while they are still receiving the data. Would it be possible to model in the 'Transfer + Processing' chart some kind of overlap of compression, transfer and decompression for streaming codecs?

A possible concrete improvement is to think that either transfer or decompression time is slowing down the decompression, not both. You could assume that the decompression starts when 10 kB of the data has been received at the specified rate, and then take the maximum of the transfer time and the transfer of first 10 kB + the decompression time.

Slower internet connections 3G, 4G, LTE, Wifi could make interesting presets: http://www.bandwidthplace.com/intern...-wins-article/

[nemequ]

Browsers (and many other transfer systems) start decompressing and processing while they are still receiving the data. Would it be possible to model in the 'Transfer + Processing' chart some kind of overlap of compression, transfer and decompression for streaming codecs?

A possible concrete improvement is to think that either transfer or decompression time is slowing down the decompression, not both. You could assume that the decompression starts when 10 kB of the data has been received at the specified rate, and then take the maximum of the transfer time and the transfer of first 10 kB + the decompression time.

Sounds like a good idea. I would have to change the labels for the segments from "Transfer" and "Decompression" to something like "Blocking" and "Streaming Decompression", but that's easy. The chart can optionally include compression time, too, so I'd have to figure out how to add that in when both compression and decompression are shown. The relevant bit of code is at squash-benchmark.js#L1145 if anyone wants to play with it; otherwise I'll try to take a look soon.

Slower internet connections 3G, 4G, LTE, Wifi could make interesting presets: http://www.bandwidthplace.com/intern...-wins-article/

I agree, but I'd rather show common speeds that people are actually seeing than theoretical maximums.

[PSHUFB]

Does the squash benchmark include any IO in the shown time for compression/decompression? Or is all The IO done out of band? Can the plugins control the block size that squash passes them?