Page 1 of 2 12 LastLast
Results 1 to 30 of 34

Thread: Rawzor, how does it work?

  1. #1
    Member
    Join Date
    Apr 2011
    Location
    Hungary
    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Lightbulb Rawzor, how does it work?

    Hello everyone,

    A couple of months ago I've ran across a lossless raw compression utility called Rawzor. I was quite impressed how it managed to compress my Olympus .orf uncompressed raws. However, the beta version has expired and the developer seems to have disappeared altogether (hopefully he didn't die...).

    I do not think the author re-invented the wheel but used a known method to compress the image data. I suspect that information is compressed either as a per-channel basis or as a proper 12 or 14-bit picture data. I am unaware of any "traditional" compression software that can recognize and/or process such data structure.

    Therefore, I'd like to know if anyone would be willing to develop a software that can compress such images. I'm not exactly rich but I would certainly donate to the project and try to raise some funds by posting at photography forums. I'm sure I'm not the only one that would be interested as disk space savings are considerable.

    Is anyone up to the challenge, or I'm just talking "meh"?

    SZGY

  2. #2
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,334
    Thanks
    209
    Thanked 1,008 Times in 533 Posts
    For now
    Code:
    Comparing files Rawzor.exe.original and RAWZOR.EXE
    000263E0: 6A C3
    Comparing files _rawzor.exe.original and _RAWZOR.EXE
    0001FEA0: 55 C3
    The compression part actually may be not so hard, and for format support there're known open-source implementations,
    but we need sample files before doing anything.
    Also it'd be more interesting if you posted some stats.

  3. #3
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Shelwien View Post
    Also it'd be more interesting if you posted some stats.
    http://www.squeezechart.com/bitmap.html

  4. #4
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    471
    Thanked 175 Times in 85 Posts
    I certainly would also donate for such a project. Since RAWZOR is discontinued, someone else should write a similar utility.

  5. #5
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    @Shelwien: Here is a sample shot of Nikon D80 raw format (NEF). It's a typical product shot for one of my customer.

    http://rapidshare.com/files/455989947/NikonD80.7z (7.87 MiB)

    As to RAW formats, all companies seem not support developer in an "easy" way. For example, I have finally downloaded Nikon SDK after filling couple of forms. SDK itself is rather big (~50 MiB). If you have problem to download it, I could upload my copy to somewhere.
    BIT Archiver homepage: www.osmanturan.com

  6. #6
    Member
    Join Date
    Apr 2011
    Location
    Hungary
    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I'll definitely post some benchmarks. Some cameras (such as Canon) perform in-camera lossless compression so those files are out, for now. dcraw is able to decompress and process such files, for example, but it can only spit out 8 or 16 bpp images.

  7. #7
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    @SZGY: Thanks for the link. I think, it's better to get that source instead of crappy SDKs. Because, according to Nikon SDK, it seems Nikon put everything in a closed-source DLL. What's the point of giving a SDK in that way!?
    BIT Archiver homepage: www.osmanturan.com

  8. #8
    Member
    Join Date
    Apr 2011
    Location
    Hungary
    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Another similar utility is Adobe's DNG Converter but unsurprisingly enough, it tends to destructify raw headers in such a way that the resulting file only works correctly in Adobe's programs. According to wikipedia, it's using lossless JPEG compression. If I remember correctly, Rawzor still managed to beat DNG. I'll be sure to include that in my benchmark.

  9. #9
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    Quote Originally Posted by osmanturan View Post
    @Shelwien: Here is a sample shot of Nikon D80 raw format (NEF). It's a typical product shot for one of my customer.

    http://rapidshare.com/files/455989947/NikonD80.7z (7.87 MiB)

    As to RAW formats, all companies seem not support developer in an "easy" way. For example, I have finally downloaded Nikon SDK after filling couple of forms. SDK itself is rather big (~50 MiB). If you have problem to download it, I could upload my copy to somewhere.
    I did some tests on the file.

    Code:
     8,256,590 NikonD80.7z
     8,614,201 NikonD80.NEF
    30,572,154 NikonD80.bmp
    45,387,990 NikonD80.png
    61,144,219 NikonD80.ppm
    61,165,432 NikonD80.tif
    43,038,029 NikonD80.tif.paq8px
    The image is 16 bit, 3 colors. I used ImageMagick to convert to bmp, png, ppm, and tif. bmp throws away the low 8 bits. All of the other formats are 16 bit. ppm and tif are uncompressed. I'm surprised that paq8px_v69 does so poorly compared to the original NEF file.

    I found some information on the NEF format at http://lclevy.free.fr/nef/
    It is actually TIFF with some kind of proprietary compression. There is some partial source code in C on the site. It looks like a simple delta + huffman code, which is why the paq8px result is so surprising. It seems to recognize the file type but thinks it's 8 bit.

    Code:
    D:\>paq8px_v69 c:\tmp\NikonD80.tif
    Creating archive c:\tmp\NikonD80.tif.paq8px with 1 file(s)...
    
    File list (23 bytes)
    Compressed from 23 to 25 bytes.
    
    1/1  Filename: c:/tmp/NikonD80.tif (61165432 bytes)
    Block segmentation:
     0           | hdr       |         8 bytes [0 - 7]
     1           | 24b-image |  30572100 bytes [8 - 30572107] (width: 11700)
     2           | default   |  30593324 bytes [30572108 - 61165431]
    Compressed from 61165432 to 43037995 bytes.
    
    Total 61165432 bytes compressed to 43038029 bytes.
    Time 5089.98 sec, used 253205784 bytes of memory
    Last edited by Matt Mahoney; 6th April 2011 at 02:32.

  10. #10
    Member
    Join Date
    Oct 2010
    Location
    New York
    Posts
    21
    Thanks
    9
    Thanked 2 Times in 1 Post
    Quote Originally Posted by Matt Mahoney View Post
    The image is 16 bit, 3 colors. I used ImageMagick to convert to bmp, png, ppm, and tif. bmp throws away the low 8 bits. All of the other formats are 16 bit. ppm and tif are uncompressed. I'm surprised that paq8px_v69 does so poorly compared to the original NEF file.
    Nikon is somewhat notorious for playing games with their RAW formats. The actual image contains 683 distinct values in each channel. A separate segment of the NEF contains a quantization curve, somewhat similar to a gamma correction that rescales the values to 16 bits when you convert it to TIFF or PPM.

    Without knowing the quantization curve, PAQ8's predictor will be sub-optimal.

    There's a decent overview of this issue here

  11. #11
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    Interesting that the quantization seems to be lost in the raw image because it seems no compressor can detect it. A bit more data.

    Code:
    40,732,658 NikonD80.tif.nz
    42,185,183 NikonD80.tif.zpaq-c3.zpaq
    42,907,146 NikonD80.tif.pmm
    43,038,029 NikonD80.tif.paq8px
    45,387,990 NikonD80.png
    46,362,921 NikonD80.tif.zpaq-c2.zpaq
    49,709,609 NikonD80.tif.7z
    50,497,068 NikonD80.tif.zpaq-c1.zpaq
    56,512,745 NikonD80.tif.gz
    61,165,432 NikonD80.tif

  12. #12
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    561
    Thanks
    212
    Thanked 200 Times in 93 Posts
    Some interesting things about compressing the .NEF file itself: There are 2 JPEG streams inside that PAQ doesn't seem to see (although they don't seem to be progressive ones, tested paq8o8 and paq8p), but Precomp does. Also, 7-Zip seems to be a bad choice to compress the file.

    Code:
    // Precomp verbose output
    Possible JPG found at position 13552, length 113413
    Best match: 113413 bytes, recompressed to 91297 bytes
    Possible JPG found at position 184960, length 830396
    Best match: 830396 bytes, recompressed to 576126 bytes
    New size: 8337882 instead of 8614201
    
    8.614.201 NikonD80.NEF
    8.337.882 NikonD80.pcf // -c-
    8.256.590 NikonD80.7z
    8.006.617 NikonD80.pcf.7z // 7-Zip, setting Ultra
    7.917.509 NikonD80.pcf_cb // -cb (bZip2)
    7.769.295 NikonD80.pcf.zpaq // zpaq 2.05 c1
    7.558.861 NikonD80.pcf.zpaq // zpaq 2.05 c2
    Last edited by schnaader; 6th April 2011 at 12:57.
    http://schnaader.info
    Damn kids. They're all alike.

  13. #13
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by schnaader View Post
    Some interesting things about compressing the .NEF file itself: There are 2 JPEG streams inside that PAQ doesn't seem to see
    I had a quick look of NEF data structure. There is a preview (thumbnail) chunk or something which includes JPEG data. But, can't comment on why they are two instead of only one.
    BIT Archiver homepage: www.osmanturan.com

  14. #14
    Member
    Join Date
    Oct 2010
    Location
    New York
    Posts
    21
    Thanks
    9
    Thanked 2 Times in 1 Post
    Quote Originally Posted by Matt Mahoney View Post
    Interesting that the quantization seems to be lost in the raw image because it seems no compressor can detect it. A bit more data.
    This goes to how the images are stored in the NEF (vs TIFF). The NEF encodes three channels (RGB) directly from the CCD. But these channels have some physical separation as the CCD can only record a single magnitude value (a tri-color grid is placed over the pixels to selectively transmit colors).

    When you are reading a RAW file of any sort, you are getting this sort of posterized data back. The file typically stores additional information about the camera settings and other sensors that can be used to create a final image. One piece of these data is Nikon's quantization curve. This is applied to each layer separately. Once that is complete, conversion software needs to determine the RGB values for each pixel in the final TIFF. Since these three values share XY coordinates in the TIFF file (but not in the NEF), the conversion software interpolates the RGB values for each pixel in the TIFF by combining multiple neighboring pixels with differing weights based on distance and color correction.

    The final result is that the full RGB bit-space may be filled in a 16-bit TIFF file, even though the raw NEF data only contains ~9 bits per pixel

  15. #15
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Seems you're talking about this: http://en.wikipedia.org/wiki/Demosaicing
    BIT Archiver homepage: www.osmanturan.com

  16. #16
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    This certainly presents some interesting opportunities in lossless image compression.

  17. #17
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    A tool like jpegsnoop should be able to find what's in there, under tools do Img Search Fwd and then step through to each chunk to see what's in there.

  18. #18
    Member
    Join Date
    Oct 2010
    Location
    New York
    Posts
    21
    Thanks
    9
    Thanked 2 Times in 1 Post
    Quote Originally Posted by osmanturan View Post
    Seems you're talking about this: http://en.wikipedia.org/wiki/Demosaicing
    Yes, thanks! I never remember that term.

  19. #19
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    561
    Thanks
    212
    Thanked 200 Times in 93 Posts
    I extracted the 2 JPEG streams. The first one is a thumbnail (570x375), the second one seems to be a full-sized JPEG (3872x2592).

    Could the full-sized JPEG be used to predict some of the bits in the raw image data, although it has artifacts and only 8 bit per component?
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	jpeg1.jpg 
Views:	208 
Size:	110.8 KB 
ID:	1527   Click image for larger version. 

Name:	jpeg2.jpg 
Views:	226 
Size:	810.9 KB 
ID:	1528  
    http://schnaader.info
    Damn kids. They're all alike.

  20. #20
    Member
    Join Date
    Apr 2011
    Location
    Hungary
    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by schnaader View Post
    I extracted the 2 JPEG streams. The first one is a thumbnail (570x375), the second one seems to be a full-sized JPEG (3872x2592).
    Could the full-sized JPEG be used to predict some of the bits in the raw image data, although it has artifacts and only 8 bit per component?
    It may not be the case here. Some cameras can take RAW+JPEG. Maybe this picture was taken in such mode? I remember there was some Sony camera that had a RAW mode where you could not turn off the embedded JPEG option, thus you ended up with huge RAW files.

  21. #21
    Member
    Join Date
    Apr 2011
    Location
    Hungary
    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Here are a few of my raws to play with.

    Canon 50D, new Canon RAW, lossless compression
    http://borg.vv.hu/raw/IMG_2546.CR2

    Canon Powershot G3, old Canon RAW, lossless compression
    http://borg.vv.hu/raw/CRW_1522.CRW

    Olympus C-8080WZ, definitely uncompressed, file sizes are always the same
    http://borg.vv.hu/raw/P4130017.ORF
    http://borg.vv.hu/raw/P4130017.ORF

    All of these files contain a small thumbnail.

  22. #22
    Member
    Join Date
    Apr 2011
    Location
    Hungary
    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts
    And now, the comparison chart!



    Here, it's clearly visible that the Canon images are already compressed, as Rawzor cannot gain that much (it's still better than Canon's algorithm). For the uncompressed images however, the compression ratio is quite impressive!

  23. #23
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    471
    Thanked 175 Times in 85 Posts
    how do you run rawzor? my beta 0.47 does not compress anymore (time trial) and it cannot be convinced using tools like runasdate.exe

  24. #24
    Member
    Join Date
    Apr 2011
    Location
    Hungary
    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Stephan Busch View Post
    how do you run rawzor? my beta 0.47 does not compress anymore (time trial) and it cannot be convinced using tools like runasdate.exe
    Just followed Shelwien's code above. You have to hexedit the executables at the given addresses. For your convenience Download

  25. #25
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,334
    Thanks
    209
    Thanked 1,008 Times in 533 Posts
    http://nishi.dreamhosters.com/v/raws_01.rar
    - all samples posted until now (img_2546.cr2 crw_1522.crw NikonD80.NEF p4130017.orf p4160025.orf)
    - dcraw.cpp modified to compile with gcc/mingw + dcraw.exe
    - test script + misc utils

    Code:
                  size     c.size  c.time d.time
    img_2546.cr2 22758895 18773437 5.703s 2.469s
    crw_1522.crw  2870834  2423159 1.219s 0.578s
    NikonD80.NEF  8614201  7039927 3.000s 1.360s
    p4130017.orf 12091780  6871873 2.156s 1.031s
    p4160025.orf 12091780  6568092 2.094s 1.000s
    Rawzor is faster than I expected and asymmetric. I guess its based on some kind of linear predictor.
    Included dcraw seems to be able to decode all of the samples (so dcraw.cpp contains all the code
    necessary to parse them).

    > Could the full-sized JPEG be used to predict some of the bits in the raw image data,
    > although it has artifacts and only 8 bit per component?

    Yes, but it would be probably faster to predict jpeg coefs based on raw.

  26. #26
    Member
    Join Date
    Apr 2011
    Location
    India
    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I developed Rawzor, and thankfully I am not dead (although I was in a near fatal road accident this week). I have been reading threads on this forum sometimes but never posted anything yet, thanks to Stephan Busch for pointing me to this thread. I can confirm that most of what is said above about Rawzor and Raw formats is close to correct.

    Shelwien, it seems you could crack the simple time check very easily. Can you please suggest a harder-to-break way to implement time check in future versions I know its impossible to have a unbreakable lock, but I will really appreciate some easy tips on making it harder to break.

    Rawzor started as a hobby project but later got ignored due to other more profitable projects I could get due to Rawzor I intend to release a new version of Rawzor with support for newer Raw formats/cameras soon but it keeps getting postponed due to random reasons. If you guys are serious about being able to donate to or fund Rawzor's development, thats a happy news

    I will release a new version sometime soon, if someone can recommend a image compression algorithm which would be fast enough and work great on more than 8-bit images (like 12 or 16 bits per sample) I will be able to improve Rawzor's compression ratio further. Matt, I wonder if your PAQ series has a fast algorithm for images.

    Sachin Garg [India]
    www.sachingarg.com | www.rawzor.com | www.imagecompression.info

  27. #27
    Member
    Join Date
    Apr 2011
    Location
    Hungary
    Posts
    8
    Thanks
    0
    Thanked 0 Times in 0 Posts
    This thread went on to discuss how to code crippleware... I'm officially sad

  28. #28
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by SZGY View Post
    This thread went on to discuss how to code crippleware... I'm officially sad
    It started with your request...

  29. #29
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,334
    Thanks
    209
    Thanked 1,008 Times in 533 Posts
    Moved to http://encode.su/threads/1269-Software-protection
    The one who started it was Sachin Garg though, not SZGY

    As to crippleware, I have more respect for people who use this approach (encryption, anti-debugger tricks etc)
    than for people who didn't bother to do it, but still throw tantrums when their program is cracked/reverse-engineered.

    Of course software protection always makes programs less convenient for users, but that's for author to decide.

    And to bring it on topic, imho the main problem with raw image compression is parsing/lossless reconstruction
    of their formats. Fortunately we have dcraw which can be used as a source of format information, but its still
    a lot of work.
    And as to compression, I don't see much choice there actually - I'd just make a plain sequential bitwise CM for it -
    speeds like 6-7MB/s (like rawzor decoding) are easily within reach like that, especially with common tricks like unary coding.
    For further speed improvement its also possible to use some kind of hierarchical coding (for example, a tree with sums of
    values in block, per color), but such transformations don't allow any compression improvements after some point,
    so afaik its better to avoid, especially when speed is ok even without that.

  30. #30
    Member
    Join Date
    Feb 2016
    Location
    Russia
    Posts
    4
    Thanks
    0
    Thanked 3 Times in 3 Posts
    Please try again "rawzor_windows_patched.7z"

Page 1 of 2 12 LastLast

Similar Threads

  1. Getting JOCL to work.
    By Piotr Tarsa in forum The Off-Topic Lounge
    Replies: 1
    Last Post: 28th October 2010, 00:50

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •