Multiple image compression problem - looking for solution

[baretomas]

Hello,

I hope this is the right forum for my question, and if not please advise me to remove the post.

I have an interesting problem. At least I think it's interesting. And I think that you might be able to help me with considering your extensive knowledge of data compression.

We have a series of images produced during the working day, around 500 000 to 800 000 of them, that needs to compressed from a BPM format, and uploaded to a storage service in the cloud (AWS is our choice, but it can be anything). The images are around 3MB uncompressed. The compression has to be lossless.

The images are all of salmon filets, so look almost identical.

I've looked at some compression technology and found the FLIF format which looks very promising for the images individually, but am wondering. If we are to send these 500 000 images to the cloud, would it be advisory to try to compress them altogether into an archive after compressing them with FLIF? Or maybe not use FLIF at all, but there are other technologies that suit this challenge better?

Any advice would be greatly appreciated!

[schnaader]

Due to the similarity of the images, I'd try to take a bunch of them (10 - 100 images), encode them as a FLIF animation (e.g. on the command line, just enter multiple input images and adjust --max-frame-lookback as you need) and see if it's better than encoding them seperately.

[Gonzalo]

I tried a few different approaches to the same problem a while ago. I haven't tried Christian's tough.
But for me, the best result was using something like this:

shar as - *.bmp | fazip rep+mm+grzip
//or
shar as - *.bmp | fazip rep+lzma

Plain and simple. That if the images are really alike. But given the amount of files you have, their sizes, and how similar they are, I would favour any solid lempel-ziv method with a large dictionary over an image-specific one. Maybe 7z, rar or even the new razor.

Also, zpaq is really great at storing similar files because it is designed to de-duplicate everything in the first place, and it can use the very first file you added to find similarities with any other you might add later. I'm a little out of my element here, so I don't know if it were to be practical to use with the cloud.

Maybe if you could provide a sample of any random day for us to try something else...


BTW:

We have a series of images produced during the working day, around 500 000 to 800 000 of them ... The images are all of salmon filets, so look almost identical.

What the fish?

Please excuse my ignorance, but would you care to elaborate on what kind of job do you have exactly? Perhaps some quality control department? Government health agency? Thank you! Just curious...

[thorfdbg]

We have a series of images produced during the working day, around 500 000 to 800 000 of them, that needs to compressed from a BPM format, and uploaded to a storage service in the cloud (AWS is our choice, but it can be anything). The images are around 3MB uncompressed. The compression has to be lossless. The images are all of salmon filets, so look almost identical.

Frankly, if it has to be lossless, you'll get 2:1 if you try something simple. If you try very hard, you get 2:1. Maybe a little bit better. But you will not get away with much less than 1.5MB. If "visually lossless" is an option, then we're back in business, but otherwise, you will not make your images considerably smaller than that.

[Jyrki Alakuijala]

I've looked at some compression technology and found the FLIF format which looks very promising for the images individually, but am wondering. If we are to send these 500 000 images to the cloud, would it be advisory to try to compress them altogether into an archive after compressing them with FLIF? Or maybe not use FLIF at all, but there are other technologies that suit this challenge better?

FLIF is great!

Psychovisually lossless: PIK's fast mode would likely be the best solution, but requires you to have AVX2 and linux. WebP -near_lossless 40 or 60 would very likely work out of the box for your needs, and is likely 20-30 % smaller than FLIF lossless. FLIF has similar near lossless abilities that are psychovisually ok.

Faster decoding -- if decoding speed with FLIF becomes a problem, you can encode/decode to WebP lossless, and get 5x faster decoding with ~2 % more bytes. For the internet lossless use size ranges (5-250 kB) WebP lossless tends to be smaller in size, too.

BCIF/PCIF seemed (benchmarking around 2011 done by others) to have wonderful lossless perfomance edge for non-resampled images, i.e., images that would include camera reconstruction artefacts, but lost that ~10 % edge if the camera resolution image was slightly downsampled (like often is in internet use).

FLIC might be great, too, but is not open source.

[Alexander Rhatushnyak]

if it has to be lossless, you'll get 2:1 if you try something simple. If you try very hard, you get 2:1. Maybe a little bit better.

Well, sometimes it's 2 : 1 with something as simple as bzip2 , but 3.5 : 1 with something not too complex, see http://qlic.altervista.org
Note the numbers in the leftmost column are clickable, and as you'll see there are actually six test sets.