JPEG Test image set made available

[thorfdbg]

...and a final last thread on a similar issue:

As part of the ICIP compression challenge, JPEG is also happy to announce that a (public subset) of its test image set has been made available. This is actually intended to be used for the ICIP compression challenge (see there), but can also be used for any image compression testing here:

http://jpeg.org/static/icip_challenge.zip

Licenses for the images are included in the zip archive. So basically, you should quote the source of the images if you use them, but for the details, look at the READMEs in the corresponding directories. Images stem from the ITU T.24 test (c) ITU, from the "Honolulu" test set (c) Olivier Pfeiffer, and the Microsoft/JPEG XR test image set, (c) Microsoft. It's only a selection of the images that have been proven useful over years of JPEG activities, due to their nature, content and statistics.

You can also get to this link though the image compression challenge:

http://jpeg.org/items/20151126_icip_challenge.html

[Alexander Rhatushnyak]

Thomas, thank you for sharing the test images!
As far as I can see, 2/14 of them are ex-JPEGs,
I'm curious whether you would replace those 2 if you knew for sure they are ex-JPEGs,
for example if you had a JPEG that
- decompresses exactly to the corresponding .ppm
- is 2 or 3 times smaller than what lossless image codecs can do (without using special algorithms for ex-JPEGs).

[thorfdbg]

Thomas, thank you for sharing the test images! As far as I can see, 2/14 of them are ex-JPEGs,

Which of them are, in your opinion, JPEGs? For some of the images, I do have "raw" versions I could process by a tone mapper.

[Jyrki Alakuijala]

Is the image acquisition technology used to obtain these images representative of what we want to use in the future?

[thorfdbg]

Is the image acquisition technology used to obtain these images representative of what we want to use in the future?

For these images, surely not. They only cover the 8bpp LDR regime, but this is what the challenge is all about - getting a "bigger picture" of the technology on the traditional JPEG "market". If you want to look at future image modalities, we'll have JPEG XT or JPEG 2000 for beyond 8bpp and HDR, and JPEG innvations / JPEG Pleno for future image modalities such as holographic compression or plenoptic (light field) images. The market for such novel ideas is, however, much smaller and one may wonder whether standardization is already needed.

[Jyrki Alakuijala]

I'm not worried about HDR. I'm worried about the artefacts in the images and if they represent the kind of artefacts we have in modern images.

Which camera models were used?

[thorfdbg]

(double post, removed)

[thorfdbg]

Which camera models were used?

The Honolulu test set was shot with a Canon EOS 400D. For the microsoft dataset, P01, P06 and P14 were shot with a Canon EOS-1DS, P04 is a sigma/foveon sensor, a Sigma SD9. P08 comes from a Canon PowerShot G5, P10 is a Fujfilm F700, P26 is a Nikon D2X. ITU did unfortunately not record in their T.24 test set how the images were acquired. Bike, woman and care are originally in CMYK, probably coming from a high-quality (back then) photo digitizer for prepress and have been converted to sRGB. Bike3 is originally in RGB, and ITU quotes its source as "Crosfield digital camera".

One may wonder why we have picked these images. The reason is that we tested multiple sets and images in subjective evaluation, and images must have features that are "interesting enough" to provoke specific artifacts, and we also need a test set that is "large enough" to cover multiple possible types of distortion. For example, it seems likely that Microsoft optimized their HDPhoto (now JPEG XR) probably a bit too much to their own collection, and it works not quite as well images outside of the expected set. "bike", "cafe" and "woman" are surely old, but they are traditional and they have interesting features - they are pretty hard to compress in various aspects and I certainly would not call them obsolete. The honolulu test set also contains a couple of natural challenging images that trigger typical defects (loss of structure in honolulu zoo, and blocking defects in the bird of paradise image). So no, these images are certainly not "picked at random".

For all the images, except for the ITU set, we do have the corresponding "raw" formats from which the PPM files have been generated. So if you have reasons to disagree with the processing of one of the images, I'll be happy to re-generate a (possibly better) image from the corresponding "digital negative" that might please you better. Actually, we can even pass them out (same license), we only need to focus on one particular set for the challenge as otherwise results will be hard to compare.

Last but not least, you are of course invited to provide your own images (four of them), provided they are given away with a license that allows publication and/or some minor editing steps such as cropping or (lossless) conversion to another format.

[Jyrki Alakuijala]

The specified camera models in the corpus and their launch years:

Canon EOS 400D -- 2006
Canon EOS-1DS -- 2002
Sigma SD9 -- 2002
Canon PowerShot G5 -- 2003
Fujfilm F700 -- 2003
Nikon D2X -- 2005

[thorfdbg]

The specified camera models in the corpus and their launch years: Canon EOS 400D -- 2006 Canon EOS-1DS -- 2002 Sigma SD9 -- 2002 Canon PowerShot G5 -- 2003 Fujfilm F700 -- 2003 Nikon D2X -- 2005

And your point is? Do you believe the image statistics are so significantly different with any later model that it renders the test unsuitable or makes any new compression engine necessary? If so, you're certainly invited to provide your own images to back up your claim. Again, we're only looking for this test at 8bpp LDR data. If you're interested in higher bitdepth or larger gamut - which does make a lot of sense - we're talking about a different game.

[Jyrki Alakuijala]

Do you believe the image statistics are so significantly different with any later model that it renders the test unsuitable or makes any new compression engine necessary?

Yes. I believe they can be significantly different, and it is not just a guess. I saw more anisotropic filtering like reconstruction artefacts in these images than I see with modern high quality images.

If so, you're certainly invited to provide your own images to back up your claim.

A high quality corpus would be an incentive for all researchers to participate.

[thorfdbg]

Yes. I believe they can be significantly different, and it is not just a guess. I saw more anisotropic filtering like reconstruction artefacts in these images than I see with modern high quality images.

Where are your images to back up the claim? Again, I'm not saying you're wrong. I'm saying, as every scientist should say, provide evidence that you're right. While details certainly differ, the variance within the image content I have seen so far are much larger than the variance due to the camera technology. Image classes and content has made so far much more a difference than the specific acquisition technology.

I'm happy to invite you to prove that this is incorrect. Provide evidence - that's what I'm saying.

A high quality corpus would be an incentive for all researchers to participate.

That's why we've picked these images. Again, they have been proven very useful already. It is not a random selection. If you made subjective tests, you will have seen that you really need images that provoke defects that are actually visible. The images here are already "filtered" in this respect. There is no "lena" or "goldhill" in this selection basically because these images are useless for any serious evaluation, unsuitable in size and content. And even though "bike" and "cafe" are old, these images *have* shown to be useful.

[Jyrki Alakuijala]

Where are your images to back up the claim? Again, I'm not saying you're wrong. I'm saying, as every scientist should say, provide evidence that you're right.

I cannot give you a scientific proof, but just to explain what I did. I zoomed your photos to 1:1 to my 1920x1200 24" monitor, and observed these artefacts:
- greens are less vivid (flatter dynamics, less variation) than with best modern methods,
- the scanned film (non-digital) photo looks over-saturated, colors are a bit cartoonish (also there seems to be a small dust particle on the sky, makes me wonder if there is more dust where it cannot be seen)
- noise is different from modern cameras -- in the proposed corpus low intensity noise is suppressed making the higher intensity noise to spike out,
- some images look more blurry than what I'm used to

Reference images can be obtained with a modern digital camera. Canon EOS 5DS (most popular camera on http://www.dpreview.com/) sample gallery works as such reference images:

http://www.dpreview.com/sample-galle...ery/8542822313

You need to look at the images at 1:1 -- or higher zoom -- to see the difference in noise patterns.

[Alexander Rhatushnyak]

Which of them are, in your opinion, JPEGs? For some of the images, I do have "raw" versions I could process by a tone mapper.

Bike3 and p06 are, as far as I can see, ex-JPEGs.

[thorfdbg]

Bike3 and p06 are, as far as I can see, ex-JPEGs.

P06 is a Canon EOS-1DS, I do have the raw version here, but I haven't processed this image to 8bit RGB. This is one of Bill Crow's images from the Microsoft dataset. It's part of the dataset because it combines several challenging aspects for compression, namely rather flat surfaces (hence, provokes blocking defects) and highly textured areas (the grill, for example, or the air filter). It seems to be too long to allow an upload here, unfortunately. )-: Just a bit over 10MB.

This is the metadata of the original:

Filename: P06-RAW.tif
Timestamp: Sun Aug 31 12:40:34 2003
Camera: Canon EOS-1DS
Owner: Bill Crow
ISO speed: 100
Shutter: 1/197.4 sec
Aperture: f/9.1
Focal length: 53.0 mm
Embedded ICC profile: no
Number of raw images: 1
Thumb size: 288 x 192
Full size: 4082 x 2718
Image size: 4082 x 2718
Output size: 4082 x 2718
Raw colors: 3
Filter pattern: GRBGGRBGGRBGGRBG
Daylight multipliers: 1.697621 0.929716 1.023688
Camera multipliers: 829.000000 512.000000 624.000000 0.000000

I do not know much about the sources of bike3 except that it comes from the ITU T.24 test set. I'm not clear now this image was processed or whether it ever was a JPEG. What makes this image interesting is the sparkle noise on the bike and the pink/turquoise contrast edge in the painting. This in particular causes interpolation errors when chroma subsampling and/or DCTs are applied to the chroma channels. It is also interesting because it is one of the images that triggered a bug in a ITU.T-851 implementation. So yes, this is a test for proper chroma processing due to the unusual coloring.

The colors in bike and cafe are really a bit weird, that's likely because these two images were originally in CMYK, and ITU does not provide information how they were converted to (some) RGB, or rather which RGB color space they are in. What's interesting about these two that they are very hard to compress due to the amount of texture, and the combination of flat and textured areas.

There is a similar image in the T.24 test set ("tools"), but this one unfortunately is subsampled in chroma, so it got removed as there are already defects in the chroma to begin with.

[Alexander Rhatushnyak]

Well, maybe p06 is a false positive, but probability that the same with bike3 is lower:

bike              1.037   NO,  this doesn't look like an ex-JPEG image.       
bike3             1.265 ! YES, this looks like an ex-JPEG image, no smoothing.
cafe              1.027   NO,  this doesn't look like an ex-JPEG image.       
woman             1.024   NO,  this doesn't look like an ex-JPEG image.       
bird_of_paradise  1.000   NO,  this doesn't look like an ex-JPEG image.       
honolulu_zoo      1.000   NO,  this doesn't look like an ex-JPEG image.       
oahu_northcoast   1.000   NO,  this doesn't look like an ex-JPEG image.       
p01               1.052   Maybe. Try better methods.                          
p04               1.018   NO,  this doesn't look like an ex-JPEG image.       
p06               1.184 ! YES, this looks like an ex-JPEG image, shifts: 5, 0.
p08               1.000   NO,  this doesn't look like an ex-JPEG image.       
p10               1.024   NO,  this doesn't look like an ex-JPEG image.       
p14               1.047   NO,  this doesn't look like an ex-JPEG image.       
p26               1.056   Maybe. Try better methods.

[Jon Sneyers]

For some lossless compression results of this test set, see row 9 of this spreadsheet: https://docs.google.com/spreadsheets...#gid=751305882
Note however that BPG looks bad in this spreadsheet because I'm using the default x265 encoder, I couldn't get the other one to compile on my platform (armv7). BPG is actually somewhat better than that.

[Jyrki Alakuijala]

For some lossless compression results of this test set, see row 9 of this spreadsheet: https://docs.google.com/spreadsheets...#gid=751305882
Note however that BPG looks bad in this spreadsheet because I'm using the default x265 encoder, I couldn't get the other one to compile on my platform (armv7). BPG is actually somewhat better than that.

Is it possible that some algorithms are run with keeping exact RGB values for alpha==0 pixels, and others clear (or manipulate) the RGB values for these pixels?

[Jon Sneyers]

Is it possible that some algorithms are run with keeping exact RGB values for alpha==0 pixels, and others clear (or manipulate) the RGB values for these pixels?

I used default lossless encoding parameters (except I used -m6 for cwebp and -m9 for bpgenc, which are the respective "maximum compression" settings), so yes, I suppose that gives a disadvantage to formats that store invisible RGB values.

For the images in this particular test set, it makes no difference since none of those images have an alpha channel.

[Alexander Rhatushnyak]

Jon, thank you for testing FLIF 0.1.10 on this test set.
BMF 2.01 compresses these images to 87255624 bytes,
so FLIF -n is at 116%, exactly as with LPCB images.

[Jyrki Alakuijala]

... I used -m6 for cwebp ...

In cwebp, you may get some gains if you use -z 9 (or both -m 6 and -q 100). If you want to explore with alpha cleanup, cwebp has a switch for it, too. -alpha_cleanup

cwebp -lossless -q 100 -m 6 -alpha_cleanup in.png -o out.png