Combining compression algorithms (image + archive)

[Noob]

Hello,

I'd like to ask the experts' opinion.

I want to distribute ~3500 bitmaps, so I'm looking to minimize the size of this bunch of files.
I used a sample of 140 bitmaps of various sizes, and ran a few experiments.

First, I just put the uncompressed bitmaps in a tar, and compressed with xz.
Then I thought I could compress the bitmaps beforehand (I tried PCX and PNG).

74M bmp
67M pcx
35M png

20M bmp.txz
24M pcx.txz
34M png.txz

As you can see, the uncompressed bitmaps weigh 74 MB.
PCX saves ~10%
PNG saves ~50%

But then compressing these folders brings disappointing results.
The BMP archive compresses best, while the PNG archive doesn't compress at all.
As a result, the compressed BMP archive is much smaller than the PNG archive.

Is this expected?

Is there a way to distribute a small archive, containing compressed images?
Maybe I should try other formats than PNG and xz?
What do you think?

Regards.

[Samantha]

You can use "bpgenc & bpgdec" compressor, in my opinion one of the best compressors for images...after converting a series of images you can to archive as you see fit.
If you need to explanation or help ask.

FOR /r "Temp\" %%F IN (*.jpg) DO (
WorkBook\bpgenc.exe "%%F" -q 30 -f 420 -c ycbcr -b 10 -o"Temp\%%~nF.bpg" "%%F"
ECHO Files Processed [%%~nF.jpg] To [%%~nF.bpg]
)
pause

bpg-0.9.6-win64.zip

[schnaader]

You could also give FLIF a try - it's specialized on images and could perform even better than BPG (and also, much better than PNG).

[Noob]

You can use "bpgenc & bpgdec" compressor, in my opinion one of the best compressors for images... after converting a series of images you can to archive as you see fit. If you need to explanation or help ask.

Hello,

Thanks for the suggestion. Does BPG have a lossless mode?

However, I think my problem statement was not very clear. The images are meant to be distributed and used as graphics resources for a game. (I'll have to check, but I don't think e.g. SDL supports BPG.)

My main question was: why do a bunch of BMP images compress to a much smaller size than the same PNG-compressed images. I think it might be because, with BMP, lots of colors repeat, but with PNG, the compression has "mixed" the bytes in a way that xz finds much less repetitions?

Regards.

[Noob]

You could also give FLIF a try - it's specialized on images and could perform even better than BPG (and also, much better than PNG).

Thanks for the suggestion. I will give it a try, to get a feel for the improvement over PNG.

I suppose a bunch of images compressed with BPG or FLIF do not compress at all with xz? (Same as with PNG).

Until now, the best compression I got was to tar the BMPs and use xz on that.
(But I don't like that solution because it takes up too much space on the target.)

Regards.

[Noob]

I'll have to check, but I don't think e.g. SDL supports BPG.

https://www.libsdl.org/projects/SDL_image/

"supports the following formats: BMP, GIF, JPEG, LBM, PCX, PNG, PNM, TGA, TIFF, WEBP, XCF, XPM, XV"

I suppose WEBP is the most recent/best algorithm in the list?

Regards.

[Intrinsic]

If you want lossless i think FLIF/PNG are your best bets overall. FLIF is not well supported yet though, there is a plugin posted here for xnview though, sadly not the awesome IrfanView yet PNGs will just work for your target audience though without having to dig up special tools they've never heard of.

TIFF with LZW compression is a good choice, and will on certain images compress better than PNG.

PCX is ancient, so avoid that.

WebP can do lossless but last i tried it wasn't that impressed but worth checking out as it was a few versions ago.

BPG can do lossless too i believe, but zero support pretty much apart from old test versions of IrfanView which he pulled due to potential copyright issues he didn't wanna risk, he said i could keep my copy for testing/personal purposes only. BPG is still based on earlier x265 code as well, not current. Good quality at low bitrates, but i would honestly still use JPEG/WEBP at higher levels anyday. 0.6.7 was released a few days ago but it just patched JS issues.

[Matt Mahoney]

To answer the original question, you have mutual information in the BMP files that goes undetected if you compress them individually.

[SvenBent]

I don't think changing to flif is going to get you near just compressing BMP's as whole. as Matt is saying by using per file compression you are only finding redundant data within a small section, by keeping them uncompressed and compressing them in a big archive you are finding redundant data through all of them.
basically it like using a small dictionary vs a big dictionary compression.

What you might try to do is use png with optimal delta filters but no compression and see if it helps. basically you want some kind of preprossessing that is not a full compression of the data.

[Jyrki Alakuijala]

74M bmp
67M pcx
35M png

Guesswork numbers:

Assuming the png is a result of a ZopfliPNG run:

If you compress with WebP lossless (or even better if with -near_lossless 60), you will get around 25.9 MB, and decoding time is unchanged.

If you compress with FLIF, you will get around 25.1 MB, but decoding gets 15x slower than with PNG and WebP lossless.

Anything else is going to be either too slow or worse in compression. If you didn't do PNG optimization yet, you will see slightly better numbers for WebP lossless and FLIF.

If you combine the images into larger images consisting of self-similar images, you will see a further improvement in compression ratio, better than your txz numbers. An image compressor can do this more efficiently than a normal compressor, particularly both WebP lossless and FLIF are more efficient than xz in compressing images. CSS sprite generation is one example application of this. See http://graphicdesign.stackexchange.c...parent-iconset

[Noob]

Guesswork numbers:

Assuming the png is a result of a ZopfliPNG run:

If you compress with WebP lossless (or even better if with -near_lossless 60), you will get around 25.9 MB, and decoding time is unchanged.

(I used imagemagick to convert from BMP to PNG.)

I had to compile cwebp 0.5 from source because my distro packages only version 0.4 (which doesn't support near_lossless option)

I used -q 100 (for max compression) for the cwebp runs.

Here are the results:

76486418 bmp
20764348 bmp.txz
36281860 png
35994088 png.txz
29079746 webp_lossless
29123488 webp_lossless.txz
18277494 webp_near_lossless_60
18314088 webp_near_lossless_60.txz

So thanks for pointing out the near_lossless option.

Regards.

[Jyrki Alakuijala]

That looks great! WebP near lossless was recently improved significantly based on butteraugli experiments, roughly so that level 40 become as good as 60 in quality, 60 as good as 80 etc. If you do a quality comparison try to get your version from the head instead of using the versioned 0.5.0.

WebP quality settings are not that simple: lossless and near lossless coding requires two additional parameters for maximum compression: -m 6 and -q 100 (or according to the help a new flag -z 9 would be sufficient, too -- but I never tried that one yet).

... and -alpha_q does not impact WebP lossless, it is only for lossy + alpha.

[SvenBent]

The problem with accepting lossy is that you can always get it smaller. i would re think if you really want to throw away quality/accuracy for the compression gains.
it all depends on what userbase it is. if its just a one time thing or a repeated compress/decompress situation. In the later you risk continual destruction of quality over time.

Just my 2 cents.

[Jyrki Alakuijala]

The problem with accepting lossy is that you can always get it smaller.

There are many aspects to it. If the loss is not-that-bad, i.e., +-1 in the most noisy areas of the images, it will be near impossible to find it, even in a delicate flip test. Still, you can get 15 % less bytes, and a 15 % speedup in decoding. cwebp -near_lossless 80 does that, with -60, it allows +-2, still near impossible to see in the noisy areas. Smoother areas are not touched.

Another cool compression option for releasing dense and fast-to-decode graphics is still the palette encoding. A lot less entropy coding can be done, and the decoding can be twice as fast as for real true color. Of course the image will suffer more than in simple -near_lossless 60 encoding.

If you are releasing a game for example, light near lossless (and palette when it works) seems to me almost always favorable to true lossless. Give us more resolution, more different textures rather than every leaf on a tree down to their correct rgb in 24 bits

Try out -near_lossless 60 and -near_lossless 80, they are not the kind of loss that you are used to with JPEG or WebP lossy.

[Noob]

WebP near lossless was recently improved significantly based on butteraugli experiments, roughly so that level 40 become as good as 60 in quality, 60 as good as 80 etc. If you do a quality comparison try to get your version from the head instead of using the versioned 0.5.0.

I had actually started by cloning the repo. But it seems the project only bundles the configure script with actual releases?

How do I perform a quality comparison? By visually comparing the two versions of the image? Or did you have objective measuring tools in mind?

Regards.

[Noob]

Try out -near_lossless 60 and -near_lossless 80, they are not the kind of loss that you are used to with JPEG or WebP lossy.

(These images are indeed game resources.)
It seems that, even though -near_lossless takes a 0..100 value, many values are equivalent?
(Apparently, only multiples of 20 are useful for now?)
Regards

[Jyrki Alakuijala]

(These images are indeed game resources.)
It seems that, even though -near_lossless takes a 0..100 value, many values are equivalent?
(Apparently, only multiples of 20 are useful for now?)
Regards

Yes, only multiples of 20 -- and most likely 0 and 20 are useless for all reasonable purposes.

[Jyrki Alakuijala]

By visually comparing the two versions of the image? Or did you have objective measuring tools in mind?

Yes, check your results visually. -near_lossless 60 should be a safe bet, and 40 is getting to a riskier zone (but likely still ok).

[Noob]

If you do a quality comparison try to get your version from the head instead of using the versioned 0.5.0.

Hello Jyrki,

Are you the dev in charge of the lossless codec in WebP?

I noticed that James Zern announced libwebp 0.5.1
https://groups.google.com/a/webmproj...ss/vdwAxnmzsbw
(God, I hate Google Groups)

Does this new version include the near_lossless improvements you had in mind?

The release notes mention:

* lossless encoding performance improvements
* memory reduction in both lossless encoding and decoding

Regards.

[Jyrki Alakuijala]

Are you the dev in charge of the lossless codec in WebP?

No. I contribute sometimes to WebP development, but I don't make decisions about any part of WebP. Particularly, I never participated in WebP lossy.

I designed the WebP lossless format in 2011, participated in the initial production-level implementation with the WebP team in early 2012, and moved away from the project in August 2012 to work on an research compressor that seeded the development of brotli.

Every now and then I improve the old WebP lossless algorithms for density or speed -- the team maintaining WebP has been focusing on WebP lossy, at least until very recently.

Near-lossless was already in my initial version in 2011, but the WebP team didn't want it in the inital production-level code. The way I implemented near-lossless is very different from the typical near-lossless coding. I don't maximize savings and quantize everywhere, but I first evaluate where the loss is invisible and try to apply it only there.

I found the near-lossless mode very useful, so I kept insisting on it, and in 2015 the WebP team was ready to add it to the production version. The most recent near-lossless improvements are a result of the development of butteraugli. I had run butteraugli on 1000 pngs for WebP near-lossless and analyzed what happened with the worst few images, and came up with the simple bankers rounding algorithm that removed the slight intensity shift that the previous version had for certain kinds of images.

Does this new version include the near_lossless improvements you had in mind?

Yes. There will be some more density/quality improvements to near-lossless, but the first big step is in 0.5.1-rc5 (and it is missing in 0.5.0).