Results 1 to 21 of 21

Thread: JPEG Test image set made available

  1. #1
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    448
    Thanks
    1
    Thanked 101 Times in 61 Posts

    JPEG Test image set made available

    ...and a final last thread on a similar issue:

    As part of the ICIP compression challenge, JPEG is also happy to announce that a (public subset) of its test image set has been made available. This is actually intended to be used for the ICIP compression challenge (see there), but can also be used for any image compression testing here:

    http://jpeg.org/static/icip_challenge.zip

    Licenses for the images are included in the zip archive. So basically, you should quote the source of the images if you use them, but for the details, look at the READMEs in the corresponding directories. Images stem from the ITU T.24 test (c) ITU, from the "Honolulu" test set (c) Olivier Pfeiffer, and the Microsoft/JPEG XR test image set, (c) Microsoft. It's only a selection of the images that have been proven useful over years of JPEG activities, due to their nature, content and statistics.

    You can also get to this link though the image compression challenge:

    http://jpeg.org/items/20151126_icip_challenge.html

  2. Thanks:

    Alexander Rhatushnyak (4th December 2015)

  3. #2
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    244
    Thanks
    43
    Thanked 103 Times in 53 Posts
    Thomas, thank you for sharing the test images!
    As far as I can see, 2/14 of them are ex-JPEGs,
    I'm curious whether you would replace those 2 if you knew for sure they are ex-JPEGs,
    for example if you had a JPEG that
    - decompresses exactly to the corresponding .ppm
    - is 2 or 3 times smaller than what lossless image codecs can do (without using special algorithms for ex-JPEGs).

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

  4. #3
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    448
    Thanks
    1
    Thanked 101 Times in 61 Posts
    Quote Originally Posted by Alexander Rhatushnyak View Post
    Thomas, thank you for sharing the test images! As far as I can see, 2/14 of them are ex-JPEGs,
    Which of them are, in your opinion, JPEGs? For some of the images, I do have "raw" versions I could process by a tone mapper.

  5. #4
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    805
    Thanks
    233
    Thanked 291 Times in 173 Posts
    Is the image acquisition technology used to obtain these images representative of what we want to use in the future?

  6. #5
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    448
    Thanks
    1
    Thanked 101 Times in 61 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Is the image acquisition technology used to obtain these images representative of what we want to use in the future?
    For these images, surely not. They only cover the 8bpp LDR regime, but this is what the challenge is all about - getting a "bigger picture" of the technology on the traditional JPEG "market". If you want to look at future image modalities, we'll have JPEG XT or JPEG 2000 for beyond 8bpp and HDR, and JPEG innvations / JPEG Pleno for future image modalities such as holographic compression or plenoptic (light field) images. The market for such novel ideas is, however, much smaller and one may wonder whether standardization is already needed.

  7. #6
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    805
    Thanks
    233
    Thanked 291 Times in 173 Posts
    I'm not worried about HDR. I'm worried about the artefacts in the images and if they represent the kind of artefacts we have in modern images.

    Which camera models were used?

  8. #7
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    448
    Thanks
    1
    Thanked 101 Times in 61 Posts
    (double post, removed)

  9. #8
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    448
    Thanks
    1
    Thanked 101 Times in 61 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Which camera models were used?
    The Honolulu test set was shot with a Canon EOS 400D. For the microsoft dataset, P01, P06 and P14 were shot with a Canon EOS-1DS, P04 is a sigma/foveon sensor, a Sigma SD9. P08 comes from a Canon PowerShot G5, P10 is a Fujfilm F700, P26 is a Nikon D2X. ITU did unfortunately not record in their T.24 test set how the images were acquired. Bike, woman and care are originally in CMYK, probably coming from a high-quality (back then) photo digitizer for prepress and have been converted to sRGB. Bike3 is originally in RGB, and ITU quotes its source as "Crosfield digital camera".

    One may wonder why we have picked these images. The reason is that we tested multiple sets and images in subjective evaluation, and images must have features that are "interesting enough" to provoke specific artifacts, and we also need a test set that is "large enough" to cover multiple possible types of distortion. For example, it seems likely that Microsoft optimized their HDPhoto (now JPEG XR) probably a bit too much to their own collection, and it works not quite as well images outside of the expected set. "bike", "cafe" and "woman" are surely old, but they are traditional and they have interesting features - they are pretty hard to compress in various aspects and I certainly would not call them obsolete. The honolulu test set also contains a couple of natural challenging images that trigger typical defects (loss of structure in honolulu zoo, and blocking defects in the bird of paradise image). So no, these images are certainly not "picked at random".

    For all the images, except for the ITU set, we do have the corresponding "raw" formats from which the PPM files have been generated. So if you have reasons to disagree with the processing of one of the images, I'll be happy to re-generate a (possibly better) image from the corresponding "digital negative" that might please you better. Actually, we can even pass them out (same license), we only need to focus on one particular set for the challenge as otherwise results will be hard to compare.

    Last but not least, you are of course invited to provide your own images (four of them), provided they are given away with a license that allows publication and/or some minor editing steps such as cropping or (lossless) conversion to another format.

  10. #9
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    805
    Thanks
    233
    Thanked 291 Times in 173 Posts
    The specified camera models in the corpus and their launch years:

    Canon EOS 400D -- 2006
    Canon EOS-1DS -- 2002
    Sigma SD9 -- 2002
    Canon PowerShot G5 -- 2003
    Fujfilm F700 -- 2003
    Nikon D2X -- 2005

  11. #10
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    448
    Thanks
    1
    Thanked 101 Times in 61 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    The specified camera models in the corpus and their launch years: Canon EOS 400D -- 2006 Canon EOS-1DS -- 2002 Sigma SD9 -- 2002 Canon PowerShot G5 -- 2003 Fujfilm F700 -- 2003 Nikon D2X -- 2005
    And your point is? Do you believe the image statistics are so significantly different with any later model that it renders the test unsuitable or makes any new compression engine necessary? If so, you're certainly invited to provide your own images to back up your claim. Again, we're only looking for this test at 8bpp LDR data. If you're interested in higher bitdepth or larger gamut - which does make a lot of sense - we're talking about a different game.

  12. #11
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    805
    Thanks
    233
    Thanked 291 Times in 173 Posts
    Quote Originally Posted by thorfdbg View Post
    Do you believe the image statistics are so significantly different with any later model that it renders the test unsuitable or makes any new compression engine necessary?
    Yes. I believe they can be significantly different, and it is not just a guess. I saw more anisotropic filtering like reconstruction artefacts in these images than I see with modern high quality images.

    Quote Originally Posted by thorfdbg View Post
    If so, you're certainly invited to provide your own images to back up your claim.
    A high quality corpus would be an incentive for all researchers to participate.

  13. #12
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    448
    Thanks
    1
    Thanked 101 Times in 61 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Yes. I believe they can be significantly different, and it is not just a guess. I saw more anisotropic filtering like reconstruction artefacts in these images than I see with modern high quality images.
    Where are your images to back up the claim? Again, I'm not saying you're wrong. I'm saying, as every scientist should say, provide evidence that you're right. While details certainly differ, the variance within the image content I have seen so far are much larger than the variance due to the camera technology. Image classes and content has made so far much more a difference than the specific acquisition technology.

    I'm happy to invite you to prove that this is incorrect. Provide evidence - that's what I'm saying.
    Quote Originally Posted by Jyrki Alakuijala View Post
    A high quality corpus would be an incentive for all researchers to participate.
    That's why we've picked these images. Again, they have been proven very useful already. It is not a random selection. If you made subjective tests, you will have seen that you really need images that provoke defects that are actually visible. The images here are already "filtered" in this respect. There is no "lena" or "goldhill" in this selection basically because these images are useless for any serious evaluation, unsuitable in size and content. And even though "bike" and "cafe" are old, these images *have* shown to be useful.

  14. #13
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    805
    Thanks
    233
    Thanked 291 Times in 173 Posts
    Quote Originally Posted by thorfdbg View Post
    Where are your images to back up the claim? Again, I'm not saying you're wrong. I'm saying, as every scientist should say, provide evidence that you're right.
    I cannot give you a scientific proof, but just to explain what I did. I zoomed your photos to 1:1 to my 1920x1200 24" monitor, and observed these artefacts:
    - greens are less vivid (flatter dynamics, less variation) than with best modern methods,
    - the scanned film (non-digital) photo looks over-saturated, colors are a bit cartoonish (also there seems to be a small dust particle on the sky, makes me wonder if there is more dust where it cannot be seen)
    - noise is different from modern cameras -- in the proposed corpus low intensity noise is suppressed making the higher intensity noise to spike out,
    - some images look more blurry than what I'm used to

    Reference images can be obtained with a modern digital camera. Canon EOS 5DS (most popular camera on http://www.dpreview.com/) sample gallery works as such reference images:

    http://www.dpreview.com/sample-galle...ery/8542822313

    You need to look at the images at 1:1 -- or higher zoom -- to see the difference in noise patterns.

  15. #14
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    244
    Thanks
    43
    Thanked 103 Times in 53 Posts
    Quote Originally Posted by thorfdbg View Post
    Which of them are, in your opinion, JPEGs? For some of the images, I do have "raw" versions I could process by a tone mapper.
    Bike3 and p06 are, as far as I can see, ex-JPEGs.

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

  16. #15
    Member
    Join Date
    Apr 2012
    Location
    Stuttgart
    Posts
    448
    Thanks
    1
    Thanked 101 Times in 61 Posts
    Quote Originally Posted by Alexander Rhatushnyak View Post
    Bike3 and p06 are, as far as I can see, ex-JPEGs.
    P06 is a Canon EOS-1DS, I do have the raw version here, but I haven't processed this image to 8bit RGB. This is one of Bill Crow's images from the Microsoft dataset. It's part of the dataset because it combines several challenging aspects for compression, namely rather flat surfaces (hence, provokes blocking defects) and highly textured areas (the grill, for example, or the air filter). It seems to be too long to allow an upload here, unfortunately. )-: Just a bit over 10MB.

    This is the metadata of the original:

    Filename: P06-RAW.tif
    Timestamp: Sun Aug 31 12:40:34 2003
    Camera: Canon EOS-1DS
    Owner: Bill Crow
    ISO speed: 100
    Shutter: 1/197.4 sec
    Aperture: f/9.1
    Focal length: 53.0 mm
    Embedded ICC profile: no
    Number of raw images: 1
    Thumb size: 288 x 192
    Full size: 4082 x 2718
    Image size: 4082 x 2718
    Output size: 4082 x 2718
    Raw colors: 3
    Filter pattern: GRBGGRBGGRBGGRBG
    Daylight multipliers: 1.697621 0.929716 1.023688
    Camera multipliers: 829.000000 512.000000 624.000000 0.000000

    I do not know much about the sources of bike3 except that it comes from the ITU T.24 test set. I'm not clear now this image was processed or whether it ever was a JPEG. What makes this image interesting is the sparkle noise on the bike and the pink/turquoise contrast edge in the painting. This in particular causes interpolation errors when chroma subsampling and/or DCTs are applied to the chroma channels. It is also interesting because it is one of the images that triggered a bug in a ITU.T-851 implementation. So yes, this is a test for proper chroma processing due to the unusual coloring.

    The colors in bike and cafe are really a bit weird, that's likely because these two images were originally in CMYK, and ITU does not provide information how they were converted to (some) RGB, or rather which RGB color space they are in. What's interesting about these two that they are very hard to compress due to the amount of texture, and the combination of flat and textured areas.

    There is a similar image in the T.24 test set ("tools"), but this one unfortunately is subsampled in chroma, so it got removed as there are already defects in the chroma to begin with.

  17. #16
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    244
    Thanks
    43
    Thanked 103 Times in 53 Posts
    Well, maybe p06 is a false positive, but probability that the same with bike3 is lower:

    Code:
    bike              1.037   NO,  this doesn't look like an ex-JPEG image.       
    bike3             1.265 ! YES, this looks like an ex-JPEG image, no smoothing.
    cafe              1.027   NO,  this doesn't look like an ex-JPEG image.       
    woman             1.024   NO,  this doesn't look like an ex-JPEG image.       
    bird_of_paradise  1.000   NO,  this doesn't look like an ex-JPEG image.       
    honolulu_zoo      1.000   NO,  this doesn't look like an ex-JPEG image.       
    oahu_northcoast   1.000   NO,  this doesn't look like an ex-JPEG image.       
    p01               1.052   Maybe. Try better methods.                          
    p04               1.018   NO,  this doesn't look like an ex-JPEG image.       
    p06               1.184 ! YES, this looks like an ex-JPEG image, shifts: 5, 0.
    p08               1.000   NO,  this doesn't look like an ex-JPEG image.       
    p10               1.024   NO,  this doesn't look like an ex-JPEG image.       
    p14               1.047   NO,  this doesn't look like an ex-JPEG image.       
    p26               1.056   Maybe. Try better methods.

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

  18. #17
    Member
    Join Date
    Oct 2015
    Location
    Belgium
    Posts
    40
    Thanks
    10
    Thanked 25 Times in 14 Posts
    For some lossless compression results of this test set, see row 9 of this spreadsheet: https://docs.google.com/spreadsheets...#gid=751305882
    Note however that BPG looks bad in this spreadsheet because I'm using the default x265 encoder, I couldn't get the other one to compile on my platform (armv7). BPG is actually somewhat better than that.

  19. #18
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    805
    Thanks
    233
    Thanked 291 Times in 173 Posts
    Quote Originally Posted by Jon Sneyers View Post
    For some lossless compression results of this test set, see row 9 of this spreadsheet: https://docs.google.com/spreadsheets...#gid=751305882
    Note however that BPG looks bad in this spreadsheet because I'm using the default x265 encoder, I couldn't get the other one to compile on my platform (armv7). BPG is actually somewhat better than that.
    Is it possible that some algorithms are run with keeping exact RGB values for alpha==0 pixels, and others clear (or manipulate) the RGB values for these pixels?

  20. #19
    Member
    Join Date
    Oct 2015
    Location
    Belgium
    Posts
    40
    Thanks
    10
    Thanked 25 Times in 14 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Is it possible that some algorithms are run with keeping exact RGB values for alpha==0 pixels, and others clear (or manipulate) the RGB values for these pixels?
    I used default lossless encoding parameters (except I used -m6 for cwebp and -m9 for bpgenc, which are the respective "maximum compression" settings), so yes, I suppose that gives a disadvantage to formats that store invisible RGB values.

    For the images in this particular test set, it makes no difference since none of those images have an alpha channel.

  21. #20
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    244
    Thanks
    43
    Thanked 103 Times in 53 Posts
    Jon, thank you for testing FLIF 0.1.10 on this test set.
    BMF 2.01 compresses these images to 87255624 bytes,
    so FLIF -n is at 116%, exactly as with LPCB images.

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

  22. #21
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    805
    Thanks
    233
    Thanked 291 Times in 173 Posts
    Quote Originally Posted by Jon Sneyers View Post
    ... I used -m6 for cwebp ...
    In cwebp, you may get some gains if you use -z 9 (or both -m 6 and -q 100). If you want to explore with alpha cleanup, cwebp has a switch for it, too. -alpha_cleanup

    cwebp -lossless -q 100 -m 6 -alpha_cleanup in.png -o out.png

Similar Threads

  1. JPEG XT new reference software, online test updated
    By thorfdbg in forum Data Compression
    Replies: 26
    Last Post: 3rd August 2015, 20:27
  2. Test JPEG Standards Online
    By thorfdbg in forum Data Compression
    Replies: 0
    Last Post: 27th November 2012, 21:14
  3. Quo Vadis JPEG - New Movements in Still Image Compression
    By thorfdbg in forum Data Compression
    Replies: 37
    Last Post: 14th June 2012, 20:47
  4. JPEG Compression Test [April 2010]
    By Skymmer in forum Data Compression
    Replies: 18
    Last Post: 7th February 2011, 22:30
  5. JPEG Compression Test [December 2009]
    By Skymmer in forum Data Compression
    Replies: 9
    Last Post: 23rd December 2009, 20:06

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •