lossless recompression of camera raw

[Stephan Busch]

Without Rawzor being updated, the compression world lacks a practical tool for lossless recompression of camera raw files.
According to Dave Coffin, author of DCRAW, such a lossless recompression could be done this way:

* Run "dcraw -i orig.raw" and printf() the "data_offset" value.
This is where the raw pixels begin.
* Figure out where the raw pixels end. This requires a little
extra logic in dcraw.c, which is not written yet.
* Copy orig.raw to meta.raw, and zero out the raw pixel data.
* Do "dcraw -E -4 -c orig.raw | tail -n +3 | dd conv=swab > pixels.raw"
IOW, dump raw pixels, cut off the header, and byte-swap.
* Compress meta.raw and pixels.raw using appropriate algorithms,
and save them in a single wrapper file. The zeroed-out raw
pixels in meta.raw will collapse to almost nothing.

To decode these images:

* Extract meta.raw and pixels.raw from the wrapper and decompress.
* Run "dcraw -I pixels.raw meta.raw". Dcraw reads the raw pixels
from pixels.raw and everything else from meta.raw.

Recreating the original raw file would be a little more
complicated, since you'd have to reverse each "load_raw" function
into a "save_raw" function. Or just wrap orig.raw instead of meta.raw,
(Adobe DNG converter allows this) but that won't save any disk space.


I am not skilled enough but I am very curious if this could be done by someone.
What is your opinion?

[namibj]

Somewhere in the DNG converter options you can uncheck the lossless compression in the DNG itself. You can then compress them with (z)paq or similar programs.
And once I, or someone else finds a way how to make a CM program ready for GPU computing I will try to create a Tool to recompress RAW's. Because the time that takes on CPU and the lack of an optimized context modell for ucompressed DNG files are the reason why I joined this forum and read hours and days about GPU computing and why I read the ZPAQ definition until I understood >90% of it. So, give me ideas where I can make the zpaq approach to CM parallel. (Working on zpaq files won't be possible, because they work bit for bit and a GPU works with 1000 threads parallel, where blocks of threads can only execute one common instruction in parallel. So no individual branching if the should be parallel.)

[Stephan Busch]

DNG is not lossless as it deletes embedded JPEG thumbnails and it applies several other things so that the output can only be decoded properly by Adobe products.
So I want to find a way around DNG.

[sh0dan]

Interesting idea. I am the author of RawSpeed, a decoder alterative to dcraw, and your idea if definitely doable, and could be done in RawSpeed with relative ease. I use a memory map for reading the files, so you could actually zero out the memory area with the image data, and just re-write the file. For most camera types it would just be a single line of code.*

I have investigated a few compression alternatives for the raw data. Delta- and zigzag-encoding the data is definitely a good thing for the data, so areas with little details have smaller values. Also, if you write 16 bit/pixel, separate high/low bytes, since the upper bytes are almost always (close to) 0. If you do that, even gzip will compress your data quite well.

Since I know quite a few raw formats - the biggest "sillyness" in terms of compression is to have everything inter-dependable, so you cannot do multithreaded decoding. Don't make the same "mistake" in your design. DNG is the only format with variable (as in Huffman) compression that can be decoded multithreaded, and they do that by separating the image into tiles. Do something similar, or have predictable "restarts", meaning pixel values that aren't predicted from other pixels.

What are you needing this for?


For supported cameras, see here: http://rawspeed.klauspost.com/supported/cameras.xml

* There is a single thing more that must be changed, but that's a small thing.

[sh0dan]

As for "restoring" that is a different matter, and much more complicated. There are three main categories of compression:

1) Lossless JPEG (Canon, Nikon, Pentax, DNG)
2) Homegrown (Olympus, Panasonic, Sony)
3) Uncompressed (Nikon Coolpix, Samsung, Sony)

Some of them use a compression curve (Nikon, Sony), and some have alternative formats (Canon mRaw, sRaw).

1) Will at least require you to store the huffman tables for the bit counter, rather complicated to replicate.
2) Depends of course, most of them are rather obscure.
3) Easily replicable, you only need to know the bit order.

So restoring is complex, and since your compressed image obviously cannot be larger (bytewise) than your original, you might even run in to trouble in some cases.

Of course, since this is "homegrown" compression most formats/cameras have their weirdnesses. Canon images for instance it split in to vertical slices, Nikon is generally a complete mess, Panasonic have their input stream split into weird blocks... the list is long.

So you are in for quite a lot of work in re-creating the images.

[Stephan Busch]

Dear sh0dan,

my major idea is to experiment with compression algorithms on the raw pixel data to find the best suitable compressor.
At this point, restoring the original is not as important.
The long-term goal is an open, royalty-free archiver for camera raw formats because raw images take much more space than they need to.
In my tests, BWT and CM compressed the original files best and now I am curious how they would behave if input wasn't already compressed.

http://www.squeezechart.com/craw.html

[sh0dan]

Made a short test, that separates a image as you described.

You can download an archive that contains the 3 canon images separated as:

image-shell.dat - the original CR2 file with the raw image section zeroed out.
image-unmodified.dat - The raw image values stored as little endian unsigned shorts

These attempts to make the image data more compressible:
image-delta-zigzag.dat -Raw data delta-encoded over 2 pixels (best on CFA images), with restart at every line. Zigzag-encoded, little endian.
image-delta-zigzag-shuffled.dat - Same as above, but most significant byte of the shorts are moved to the end of the file.

Download it from here: http://download.klauspost.com/rawcompress/

Zigzag-encoding basicly converts signed shorts to values that are closer to zero for negative numbers: http://stackoverflow.com/questions/4...igzag-encoding

static __inline ushort16 ZigZag(short word) {
  return (word >> 15) ^ (word << 1);
}

static __inline int ZigZagReverse(ushort16 u) {
  return ((int)(u >> 1)) ^ ((int)(-(u & 1)));
}

I tried it out on some of the supported images.

Here are the combined filesizes, when the shell+raw image are put in to a zpaq archive (6.41, method 4)

24 392 279 canon_eos_5d_mark_iii_05.zpaq
25 222 506 canon_eos_6d_14.zpaq
23 708 698 canon_eos_m_04.zpaq
11 939 259 nikon_1_v2_17.zpaq
18 565 387 nikon_d4_10.zpaq
29 650 408 nikon_d5200_14.zpaq
16 202 477 olympus_epm2_16.zpaq
15 931 602 olympus_om_d_e_m5_24.zpaq
12 110 333 olympus_xz2_10.zpaq
19 684 345 pentax_k5_ii_12.zpaq
14 394 578 pentax_q10_19.zpaq
22 689 891 samsung_nx1000_19.zpaq
25 581 692 samsung_nx20_01.zpaq
14 256 129 sony_a55.zpaq
19 904 270 sony_a77_08.zpaq
19 799 467 sony_a99_04.zpaq

Most of them beat all, but the DNG versions.

If you want to do better than DNG, the biggest difference will be in removing the JPEG preview inside most of the RAW files.

[Stephan Busch]

The results are already pretty impressive in my opinion.
Would you please provide an executable that does this decoding and separating?
The Zigzag-Encoding is also interesting and seems to suit well for this purpose.

I wonder why DNG is still better as it uses lossless JPEG (Huffman) only.
There might be gain however if the JPEG thumbnails are deleted.
A smarter solution might be recompressing them losslessly.

[sh0dan]

I added a win32 binary (and the modified source) at the URL above.

Syntax is "RawSpeedCompress rawfilename". I added detection for some camera makes. I know at least Panasonic doesn't work, but there may be others where it doesn't delete the correct data.

There will be a few differences in the "shell" compared to the original. That is because Rawspeed does some in-place endianess-swapping for short/integer arrays - I didn't bother to refactor that - it shouldn't have any impact on compression.

As for DNG, it seems like the files were converted with no JPEG preview. Here are the sizes using DNG Converter v7.4:

24 583 236 canon_eos_5d_mark_iii_05-full.dng
22 494 700 canon_eos_5d_mark_iii_05-medium.dng
22 418 016 canon_eos_5d_mark_iii_05-none.dng

Cameras usually have a static huffman table for their images. The DNG converter doesn't have the same limitation - that may give it a few percent. Futhermore the DNG is compressed in tiles, which may also help it to optimize the huffman codes further. But the big difference is in the JPEG preview.

[sh0dan]

Here are the DNG sizes with a large JPEG preview:

24 583 236 canon_eos_5d_mark_iii_05-full.dng
25 596 106 canon_eos_6d_14-full.dng
23 447 770 canon_eos_m_04-full.dng
12 516 132 fujifilm_finepix_x100_11-full.dng
15 731 802 fujifilm_xf1_08-full.dng
17 263 738 fujifilm_x_e1_20-full.dng
 9 152 228 leica_m82_05-full.dng
12 743 676 leica_x1_10-full.dng
12 025 368 nikon_1_v2_17-full.dng
18 647 024 nikon_d4_10-full.dng
27 251 952 nikon_d5200_14-full.dng
21 263 236 olympus_epm2_16-full.dng
21 297 624 olympus_om_d_e_m5_24-full.dng
18 630 278 olympus_xz2_10-full.dng
19 114 972 panasonic_lumix_dmc_gh3_10-full.dng
18 049 280 panasonic_lumix_g5_15-full.dng
22 115 818 pentax_k5_ii_12-full.dng
13 851 760 pentax_q10_19-full.dng
19 992 364 samsung_nx1000_19-full.dng
22 493 996 samsung_nx20_01-full.dng
13 933 500 sigma_dp2-full.dng
21 616 376 sony_a55-full.dng
30 072 016 sony_a77_08-full.dng
26 851 270 sony_a99_04-full.dng
   468 241 522 bytes

[Stephan Busch]

Most of the camera raw in my testset seem to have 2 embedded JPEG thumbnails.
My idea is to extend your test but compress the 'image-shell.dat' with Precomp 0.4.3 (commandline -cn)
and then compress the image-delta-zigzag-shuffled.dat and the Precomp output with p.ex. ZPAQ.

The tiles idea is really good because it speeds up compression and decompression.
I will redo DNG tests.

In the meantime, I have put online an updated test-sheet with your results: http://www.squeezechart.com/craw2.html

Is there a chance for future RawSpeed, to support the camera files that are currently not supported?

[sh0dan]

I see this mostly as an academic exercise, still now convinced it has any real-world application. However, as a test of concept it works ok.

The delta-left "compression" is purely arbitrary. Just for fun, I tried making it delta-down (leftmost 2 pixels) and delta-left for the rest. Surprisingly, that gave a nice size reduction.

* Downwards delta-encoding used on border pixels.
* Added Panasonic support.
* Don't apply linearisation tables on Nikon and DNG's. The linearisation should be re-producible from the "shell"-data.
* DNG's no longer cropped, so now retains all data.

That gives the following sizes (fuji+sigma are simply zpaq encoded):

24 387 048 canon_eos_5d_mark_iii_05.zpaq
25 217 125 canon_eos_6d_14.zpaq
23 705 446 canon_eos_m_04.zpaq
12 434 658 fujifilm_finepix_x100_11.zpaq
14 425 321 fujifilm_xf1_08.zpaq
12 792 840 fujifilm_x_e1_20.zpaq
 5 920 202 leica_m82_05.zpaq
10 453 231 leica_x1_10.zpaq
10 698 628 nikon_1_v2_17.zpaq
18 566 863 nikon_d4_10.zpaq
26 185 740 nikon_d5200_14.zpaq
16 197 954 olympus_epm2_16.zpaq
15 924 670 olympus_om_d_e_m5_24.zpaq
12 102 619 olympus_xz2_10.zpaq
14 026 175 panasonic_lumix_dmc_gh3_10.zpaq
15 236 955 panasonic_lumix_g5_15.zpaq
20 012 584 pentax_k5_ii_12.zpaq
14 595 154 pentax_q10_19.zpaq
22 691 159 samsung_nx1000_19.zpaq
25 582 527 samsung_nx20_01.zpaq
14 383 003 sigma_dp2.zpaq
45 871 260 sigma_sd1_merrill_13.zpaq
14 244 370 sony_a55.zpaq
19 892 176 sony_a77_08.zpaq
19 796 622 sony_a99_04.zpaq
   455 344 330 bytes

Also tried adding precomp. Sizes with "precomp -c- -t+j -d0" on shell file:

23 907 098 canon_eos_5d_mark_iii_05.zpaq
24 423 897 canon_eos_6d_14.zpaq
23 002 821 canon_eos_m_04.zpaq
12 256 927 fujifilm_finepix_x100_11.zpaq
14 251 039 fujifilm_xf1_08.zpaq
12 667 259 fujifilm_x_e1_20.zpaq
 5 920 237 leica_m82_05.zpaq
10 451 933 leica_x1_10.zpaq
10 224 204 nikon_1_v2_17.zpaq
18 029 156 nikon_d4_10.zpaq
25 288 018 nikon_d5200_14.zpaq
15 950 513 olympus_epm2_16.zpaq
15 709 729 olympus_om_d_e_m5_24.zpaq
11 839 243 olympus_xz2_10.zpaq
13 886 785 panasonic_lumix_dmc_gh3_10.zpaq
15 121 423 panasonic_lumix_g5_15.zpaq
19 620 423 pentax_k5_ii_12.zpaq
13 860 319 pentax_q10_19.zpaq
21 441 988 samsung_nx1000_19.zpaq
24 084 539 samsung_nx20_01.zpaq
14 050 816 sigma_dp2.zpaq
44 202 248 sigma_sd1_merrill_13.zpaq
14 100 595 sony_a55.zpaq
19 722 748 sony_a77_08.zpaq
19 671 905 sony_a99_04.zpaq
   443 685 863 bytes

I updated the executable and the source at http://download.klauspost.com/rawcompress/

I don't expect to be able to throw much more time at this, but if you find someone who is interested in picking it up, I'd be happy to assist.

[packDEV]

Hi, just a quick suggestion: Although ZigZag in general is a very good means of converting 2-dimensional data to 1-dimensional for compression, there's something that leads to better results in every case tested (at least for me), called "Hilbert scan". See here: http://en.wikipedia.org/wiki/File:Hilbert_curve.svg

I used this in early versions of packJPG (before 2-dimensional context modeling), too. The Hilbert curve can be expanded for any rectangular (non square, too) format by just concatenating multiple Hilbert curves. I could even provide the source code (from an earlier version of packjpg) to calculate it. Just let me know if you are interested.

Also (second suggestion): Most of these cameras use a standard Bayer pattern: http://en.wikipedia.org/wiki/Bayer_pattern
It could make sense to differentiate between red, green and blue pixels when calculating delta, but one must keep in mind that it must be known which pixels are which.

[Stephan Busch]

I have tested RawSpeedCompress v2 now and updated results:

http://www.squeezechart.com/craw2.html

[thorfdbg]

Without Rawzor being updated, the compression world lacks a practical tool for lossless recompression of camera raw files. I am not skilled enough but I am very curious if this could be done by someone. What is your opinion?

First, you should understand that "camera raw" isn't as raw as you might believe. What you get there are not unprocessed sensor data, but something the camera vendor sells as "raw". Which amount of preprocessing has been done on such data is vendor dependent, but it's more than "nothing". You most likely get not un-mosaik'd data, you probably get more bits per pixel, but that's very much about it. The trouble is that vendors are not at all open concerning their format, and while dcraw is capable of interpreting a lot of them, they are all processed in one way or another - there's always some implied loss already when you get the data. What you can do is collect as much data as you can get, and hope that you don't loose too much on the way - that's very much where DNG is.

[thorfdbg]

Hi, just a quick suggestion: Although ZigZag in general is a very good means of converting 2-dimensional data to 1-dimensional for compression, there's something that leads to better results in every case tested (at least for me), called "Hilbert scan". See here: http://en.wikipedia.org/wiki/File:Hilbert_curve.svg

Been there, done that. We made a lot of experiments with a (luckely no longer existing) wavelet codec about 15 years ago and tried various scan patterns, and the Hilbert curve was one of them. Believe me, it doesn't make much of a difference once you are in a space where data is already almost decorrelated. Back then, that was a wavelet HH pass. Scan patterns made no difference. The part where it becomes interesting is the decorrelation and the probability model.

Also (second suggestion): Most of these cameras use a standard Bayer pattern: http://en.wikipedia.org/wiki/Bayer_pattern It could make sense to differentiate between red, green and blue pixels when calculating delta, but one must keep in mind that it must be known which pixels are which.

Something you almost surely want to do, knowing the Bayer pattern, of course. That, plus a spatial decorrelation. Probably for your interest, early versions of JPEG XR aka HDPhoto had a Bayer pattern compression. It never made it into the final standard more or less because camera vendors were opposed to this idea. Opening up how your Bayer pattern is designed is for them opening up their technology. It's pretty much an uphill battle.

[Stephan Busch]

Of course, camera vendors are not interested in opening up their formats and the weak compression has two advantages: The pixel data is compressed really fast and to the normal user big files stand for much pixels and detail. However, I don't see this as an uphill battle. It is an interesting and educational project that allows us to further push the compresion boundaries we are currently facing.

[nburns]

Of course, camera vendors are not interested in opening up their formats and the weak compression has two advantages: The pixel data is compressed really fast and to the normal user big files stand for much pixels and detail. However, I don't see this as an uphill battle. It is an interesting and educational project that allows us to further push the compresion boundaries we are currently facing.

It sounds like a fun project. A few years ago I played with lossless audio formats. I didn't mess with the actual samples, but I set out to make a format for storing an exact copy of a CD. I thought I'd have to engineer a container format, but as it turned out, the data was basically ~72 minutes worth of audio + simple TOC metadata. Flac was able to store all the metadata already, just behind an inconvenient interface and with poor support from client applications. I wrote scripts and archived all my CDs to this format. Later on, I discovered that wavpack and its client apps did a better job with the metadata (with a trivial sacrifice in compression).

I suspect that the weak compression in raw files is due mainly to the cost of the circuitry needed to do the compression, and the limited importance of raw files as compared to jpeg and other standard formats. Since each company hacks their own .raw, it's obviously not a priority to highly engineer this format, and the lightweight compression sounds like it's done in ASICs. The real purpose of this compression may be to maximize throughput between the internal components so you can keep clicking photos, rather than to make smaller files. Once the data is off the critical path, it's probably meant to be further processed on a microcontroller running firmware that outputs jpegs or whatever.

[nburns]

DNG is not lossless as it deletes embedded JPEG thumbnails and it applies several other things so that the output can only be decoded properly by Adobe products.
So I want to find a way around DNG.

DNG probably needs to be somewhat lossy, because it's supposed to be universal. It sounds like what's in .raw files is at least partly a function of what parts happen to be in the camera and what type of unprocessed data they generate, without added abstraction, like a core dump. To make a format like DNG that abstracts away the camera, you probably have to do some amount of lossy processing, like dealing with sensor geometries that are unique to some SSD chip and making them into rectangular pixels.

Edit: I think other posters already basically said as much, but using expert-level tech-speak, so perhaps it's worth being rephrased by non-experts like me.

[nburns]

I have investigated a few compression alternatives for the raw data. Delta- and zigzag-encoding the data is definitely a good thing for the data, so areas with little details have smaller values. Also, if you write 16 bit/pixel, separate high/low bytes, since the upper bytes are almost always (close to) 0. If you do that, even gzip will compress your data quite well.

XOR would probably work as well as delta + zigzag, as long as you're not expecting the resulting numbers to be meaningful. I think you could prove that the number of 1-bits this generates is optimal if the original sequence is not oscillating around a particularly poor central value (centering on zero is the worst possible with signed integers, hence the need for zigzag).

It might help to combine the numbers that make up a pixel into a single number, using something like a Morton code. This ought to help general purpose algorithms like LZ and the BWT stay centered on pixel boundaries, rather than treating everything as a random soup of uncorrelated numbers, where red=45 could predict green=12 in one place, and green=45 could predict blue=12 somewhere else, based on the same pattern. The BWT should do an okay job with multi-byte integers as long as 1 integer = 1 pixel.

Of course, the other problem is that the samples have uncompressible noise. The ideal solution would probably be to somehow filter out the noise and store it as an uncompressed stream, and compress the rest. Since the noise would tend to occupy the low bits of the samples, that's approximately what you're doing by segregating the bytes.

[sh0dan]

ok, guys - the challenge for you is to find the best algorithm that suits the data.

Here is a (log10) plot of the values of a typical image: http://download.klauspost.com/rawcom...elta-canon.txt


We CAN have values from 0 to 65535 (unsigned short), but values are mostly close to zero. 63.6% are in the 100 lowest values, 97% within 1000, 10321 unique values.

Zpaq method 4 compresses this to 8.2 bit per pixel, 7zip/gz are close, but slightly worse.

So - which compression type would be optimal for this type of data?

If you want to play with the data ifself, here it is: (unmodified / shuffled)

PS. Don't assume this distribution - there can be "gaps", like this high ISO Canon image, which can be exploited: http://download.klauspost.com/rawcom...n-HIGH_ISO.txt

[Stephan Busch]

I would be interested in testing RawSpeedCompress with Hilbert-curves. In my tests on not modified and uncompressed raw, BWT and CM outperform
every LZ-based variant.

[thorfdbg]

Well, even if I'm afraid to spoil the fun a bit, image compression is not "finding the best scan over the data", but rather finding a good image model. LZ, BWT and frieds are not the right approach here since the data you got is not "human generated" but consists mostly of noise, though with known statistics (see below). The principles you need are "decorrelation" and "energy compaction", with a suitable statistical model. JPEG-LS is a nice starter. Try to combine that with a better cross-component correlation, make use of the position of the samples of the Bayer patterns to improve the prediction, probably try a non-local matching of patterns. You probably also want to read the Weinberger paper on JPEG LS which describes a lot of the techniques, and in specific one important property of natural data prediction residuals: These are almost always two-sided Laplacian (or close to that), a family of probability distributions "stable under convolution". The reason why LS works so nice is because it makes use of this property. In the end, you'll likely notice that with a pretty primitive prediction, you get about 1:2 compression. With something very sophisticated, you get a bit above 1:2. (-; Reason is that most data you got there is noise, and not actual image data, and noise cannot be compressed very well. It's Laplacian noise, though.

[Stephan Busch]

This is getting more and more exciting and interesting.

[sh0dan]

Yeah - my thoughts mainly went towards having in-frame block matching, so say an 8x8 block could reference another (already decoded) part of the image, and only deltas between the "referenced" part and the current block was encoded - sort of like MPEG motion vectors, except inside the same image - it could also have an optional offset/scale value, so changes in lightness could be corrected for.

Might bring 5-10% or so, but I wouldn't expect much more, since it will not help with noise and high-detail areas are unlikely to be repeated.


PS. Tried sorting lines by sum of all delta-values for the line before writing it to a file, but the cases I tried with that resulted in worse compression.

[nburns]

PS. Tried sorting lines by sum of all delta-values for the line before writing it to a file, but the cases I tried with that resulted in worse compression.

This data may be inherently a bad match for the BWT, so to the extent that the BWT is working at all, it's not working the way it works on text (if the BWT or something like it is what you're going for).

In general, though, I don't think you'd want to apply deltas ahead of sorting. Sorting matches like with like, and then you can do deltas after.

I'm sure thorfdbg is right that BWT/LZ is the wrong approach to compressing this data. I already had a strong suspicion that this was a well-studied problem, and it would be more useful to study the literature than to try to invent something straight away.

[Stephan Busch]

General purpose compression might be the wrong approach here, but history shows that most scientific inventions were done with the help of experiments.
BWT and CM were best-performing on the unmodified, non-splitted Raw, and we have nothing else to test here at present.

If someone would provide RawSpeedCompress with Hilbert-curves or XOR instead of Delta + ZigZag, we could test how much gain they might bring.
JPEG-LS is also a good idea. I am wondering if a compression ratio of 1:2 could really be made real.

[thorfdbg]

Allow me to elaborate: BWT and LZW work so well on program data and text because such data is almost always based on "phrases": For natural language, these are words, and words are put into sentences. Same idioms appear all over, and the automatic dictionary built-up by LZW identifies these phrases or words. Same goes for programs: Here, the compiler uses specific "phrases" of machine language, you use specific "language idioms" when designing a program. All that can be picked up and made use of by a good compressor. The same holds *not* true for images. Here, nothing matches "perfectly", you only find approximate matches. The principle is here: Find a match that is good enough, then encode the residual. As I already said, "residuals" are almost always Laplacians (if you do it right). This implies that you can gain not very much by a smart "predictor" or a smart "dictionary" as you do in text or program compression. No matter how hard you look, you will not find an exact replicate. You can predict (from the neighbourhood) or "compact" the energy in a few coefficients, but you're always left with a residual you missed. In fact, as you will find out, for *lossless* compression this "residual" is actually the dominant part. Lossy compressions work so well because they have a good model how to distinguish "residual" from "data", and then "throw the residual away". More or less. Last but not least: 1:2 is entirely feasible and not hard to reach. Nevertheless, you can learn a lot from doing that. As you'll find out, LZW and BWT is *not* the way to go (go, try yourself, you'll see). It's a completely different business. (A fun one I'm following for years, of course...)

[nburns]

The principle is here: Find a match that is good enough, then encode the residual. As I already said, "residuals" are almost always Laplacians (if you do it right). This implies that you can gain not very much by a smart "predictor" or a smart "dictionary" as you do in text or program compression. No matter how hard you look, you will not find an exact replicate. You can predict (from the neighbourhood) or "compact" the energy in a few coefficients, but you're always left with a residual you missed.

That's kind of interesting. The BWT might not do a bad job of grouping similar data points based on these weak predictions. But to encode the output, you might need a different approach than you'd use for text, because values would be near each other, but not exact matches.

In fact, as you will find out, for *lossless* compression this "residual" is actually the dominant part. Lossy compressions work so well because they have a good model how to distinguish "residual" from "data", and then "throw the residual away". More or less. Last but not least: 1:2 is entirely feasible and not hard to reach. Nevertheless, you can learn a lot from doing that. As you'll find out, LZW and BWT is *not* the way to go (go, try yourself, you'll see). It's a completely different business. (A fun one I'm following for years, of course...)

LZW and BWT might be the way to go if you'd rather take the easy route and not spend too much time on it. DNG must be doing basically that, because its file sizes are similar to what LZW and BWT give you.

Actually, from a practical standpoint, using simple compression could be useful in this situation, because it could help you to support a wide range of .raw formats without having to create a different version for each camera.

[nburns]

If someone would provide RawSpeedCompress with Hilbert-curves or XOR instead of Delta + ZigZag, we could test how much gain they might bring.

XOR is unlikely to compress better. It might speed things up a tad.