unpackmp2 - a preprocessor for lossless compression programs

[smack]

unpackmp2 is a filter program that performs a lossless transformation of MPEG audio Layer II data (mp2) into an unpacked format and back into mp2 format.

It is intended to be used as a preprocessor for lossless compression programs in order to achieve a better compression of mp2 files.

The attached archive contains Windows executables and scripts. Also included is the full source code. It should compile under Linux and other systems as well. This is free software under GNU GPL v3, http://www.gnu.org/copyleft/gpl.html

test results
------------
test1.mp2 (125.8 MiB, 54min 57sec, 320kbps, 48kHz, stereo, source: DVB-S radio)

program            options            compressed size     unpackmp2 gain
-----------------  -----------------  ------------------  ---------------------
unpackmp2 | lpaq8  5                  110859854 = 84.0%   12489717 (11.9 MiB)
unpackmp2 | 7-Zip  7z LZMA ultra      117928435 = 89.4%   8262710  (7.9 MiB)
unpackmp2 | 7-Zip  7z Bzip2 ultra     119080485 = 90.3%   7493074  (7.1 MiB)
unpackmp2 | 7-Zip  7z PPMd ultra      120598552 = 91.4%   4933122  (4.7 MiB)
unpackmp2 | 7-Zip  Zip Deflate ultra  120718098 = 91.5%   6127333  (5.8 MiB)
lpaq8              5                  123349571 = 93.5%   (n/a)
7-Zip              7z PPMd ultra      125531674 = 95.2%   (n/a)
7-Zip              7z LZMA ultra      126191145 = 95.7%   (n/a)
7-Zip              7z Bzip2 ultra     126573559 = 96.0%   (n/a)
7-Zip              Zip Deflate ultra  126845431 = 96.2%   (n/a)
(original mp2)     (n/a)              131912640 = 100.0%  (n/a)
unpackmp2          (n/a)              202374495 = 153.4%  -70461855 (-67.2 MiB)

[Shelwien]

Nice one, thanks.
Although it'd be better if you could include a small sample for testing.

Also, we had a thread about similar mp3 tool before:
http://encode.dreamhosters.com/showp...0&postcount=19

[Skymmer]

Thank you smack!
But, oo-oof, I'm still dreaming about BIK preprocessor

[smack]

Although it'd be better if you could include a small sample for testing.

mum.mp2 (19.77 seconds, 320 kbps, 48 kHz, stereo, source: DVB-S radio)

program               compressed size  
-------------------   -----------------
unpackmp2 | lpaq8 5   676317  = 85.5%
lpaq8 5               764512  = 96.6%
(original mp2)        791040  = 100.0%
unpackmp2             1288770 = 163.0%

Of course, you could always encode mp2 files yourself... (using TwoLAME, Win32 binaries at RareWares.org, for instance)

Also, we had a thread about similar mp3 tool before:
http://encode.dreamhosters.com/showp...0&postcount=19

Yeah, I had found that one before. Do you have plans to develop it further or to open its source? I know that an "unpackmp3" program would be much more interesting than unpackmp2 for most people...

[Shelwien]

> Of course, you could always encode mp2 files yourself...
> (using TwoLAME, Win32 binaries at RareWares.org, for
> instance)

Yeah, but there's always risk of making a mistake with
unfamiliar format, so imho its better to have a known sample.
Also, even within the same format its frequently possible
to acquire different results with different coding options.
Actually we discarded the layer-I-II parsing code just
because we haven't been able to find any "live" samples to test with.
While being able to artifically produce some files doesn't mean
that there's any sense in supporting their recompression.

> Also, we had a thread about similar mp3 tool before:
> http://encode.dreamhosters.com/showp...0&postcount=19

> Yeah, I had found that one before.
> Do you have plans to develop it further or to open its source?

Its kinda further developed in http://soundslimmer.com

And as to source, I kinda planned to open it when started that
thread, but it seems like there's no demand for that
(at least nobody asked until now), and that source requires
some cleaning (like replacing the unreasonable built-in
huffman tree compression with static trees .
But I can send you a copy if you want.

> I know that an "unpackmp3" program would be much more
> interesting than unpackmp2 for most people...

Who knows. AC3/DTS are required for DVD/BD recompression
and AAC/ogg for mp4/mkv recompression, and HEAAC for stuff
like youtube.
So mp3 might be already becoming a history too.

[smack]

Actually we discarded the layer-I-II parsing code just
because we haven't been able to find any "live" samples to test with.
While being able to artifically produce some files doesn't mean
that there's any sense in supporting their recompression.

Well, you can call mp2 out-dated because it's 20 years old and a lot of more advanced codecs exist now. But it's not dead, you can find samples easily: it's used in DVD-Video (although many discs use AC3/DTS) and DVB for terrestrial and satellite TV/radio broadcast. The latter is my mp2 data source.

But I can send you a copy if you want.

Yes please, I would like to have a look at it.
There is another interesting mp3 transformation tool called MP3packer. It re-arranges mp3 data with optional brute-force Huffman optimization in order to create smaller mp3 files. It's open source but written in OCaml.

Who knows. AC3/DTS are required for DVD/BD recompression
and AAC/ogg for mp4/mkv recompression, and HEAAC for stuff
like youtube.
So mp3 might be already becoming a history too.

Is there a new SoundSlimmer in the works that will compress those formats?

Anyway, I'm not interested in "market shares" when I think about what to program in my spare time. At the moment I'm still thinking about improving mp2 recompression. Do you have any ideas how to transform the scale factors and quantized samples to make them more compressible for LPAQ, LZMA etc?

[Shelwien]

> Yes please, I would like to have a look at it.

Sent via PM.

> There is another interesting mp3 transformation tool
> called MP3packer. It re-arranges mp3 data with optional
> brute-force Huffman optimization in order to create
> smaller mp3 files. It's open source but written in OCaml.

Afair there was also a version written in perl.
Well, my parser can do something similar too.

> Is there a new SoundSlimmer in the works that will
> compress those formats?

Probably not soundslimmer as I don't have any relation
to its current copyright holders.
But I'm working on a project where such recompression
stuff would be eventually required.

> At the moment I'm still thinking about improving mp2
> recompression.

I'd suggest to think about making it completely lossless
first. Otherwise it gets very hard to prove that modified
version is equivalent to original, and its completely
incompatible with stuff like torrents.

> Do you have any ideas how to transform the scale factors
> and quantized samples to make them more compressible for
> LPAQ, LZMA etc?

Forget LZMA as you can't depend on frequently appearing strings
for compression of such data.
Also, the data in question consists of a few 2D tables,
so the applicable existing models would be mostly the ones used
for image compression, like BMF or paq8 bmp model
(in fact it was demonstrated in the mp3dump thread).

And as to usual CM/PPM coders, these are not very good for
this case, because they never do anything to the data which
they use as a context. The same data is encoded and then
used as context, and only sequentially.
So while I can't say that reaching better compression by
preprocessing is impossible, but to acquire good results
you'd have to use some kind of statistical ranking model,
so its imho much more reasonable to just write a specialized
coder right away.

In short, you can't depend on preprocessing too much -
btw that's why paq has different submodels for different
data types.

[Shelwien]

Well, I can suggest something (though maybe you already do it).
1) It should be better to store the coefs by columns instead of rows.
2) Element width matters a lot - its usually better to use a bytewise
code, but it might be different if you're willing to use a wav model or something.
3) Starting with simple delta (storing coef differences instead of original coefs),
its usually possible to gain something by predicting the next value and
subtracting the prediction before coding.

[foxyshadis]

Well, I can suggest something (though maybe you already do it).
1) It should be better to store the coefs by columns instead of rows.

Not columns - zigzag, as they're natively stored in most formats. Don't know about MP2. Statistical analysis can show you the optimal zigzag; JPEG uses a very basic diagonal one, whereas H.264's jumps all over the place. As you evolve your prediction model you'll probably have to run the analysis a few more times. There might be an easy way to signal an End Of Block, for low bitrates, I don't know enough about MPEG-1-L-II to say. Either give each block a length or define a magic byte.

3) Starting with simple delta (storing coef differences instead of original coefs),
its usually possible to gain something by predicting the next value and
subtracting the prediction before coding.

It's much better to extend this inter-frame. Start by storing every DC component from the entire block next to each other. You should be able to predict better than simple delta by using lossless audio techniques: Predict it by fitting a curve to the last x samples. Then store the AC components progressively, each band grouped together. Now you can try to predict it from the preceding coeffs, I suppose, but I'm sure you'll need to go through a lot of models before you find a good predictor. Near the end some will be shorter because the blocks were terminated early in the first step; generally this seems to save more than leaving the zeros in.

A context mixer might not be a bad choice at this stage. It might even point to further exploitable patterns. The packjpg paper and the h.264 standards both have a lot of very good information on treating dct-quantized data, they're worth an overview. (I'm far more familiar with those formats and what works than MP2.)

Note that this method prevents any kind of real-time playback. The larger the blocks, the more data has to be decompressed before playback.

[smack]

I'd suggest to think about making it completely lossless
first. Otherwise it gets very hard to prove that modified
version is equivalent to original, and its completely
incompatible with stuff like torrents.

It's lossless when you compare the decoded audio data of the original and the unpacked/repacked mp2 files, using foobar2000's "Bit-compare tracks" feature or by simply "fc"-ing the decoded wav's.

The program currently skips all non-audio data in the mp2 files - that's a feature not a bug. But I recognise your point that for most uses it would be better to preserve the original mp2 with all its non-audio data, such as ID3 tags at head or tail of file and the "filler" data inside of the mp2 frames. I'm going to add that to the next version.

1) It should be better to store the coefs by columns instead of rows.

I think unpackmp2 already does this: it orders the scale factors and samples by subband and groups many consecutive frames in order to create longer runs of similar bytes. This is a kind of transforming the "2D tables" of data into "1D streams", isn't it?

2) Element width matters a lot - its usually better to use a bytewise
code, but it might be different if you're willing to use a wav model or something.

Yes, I noticed this at the very beginning of my experiments. Unpackmp2 "inflates" nearly all data elements of the mp2 stream to full bytes. This is the reason why the unpacked files are so much larger than the mp2 files. It seems to help compressors quite a lot.

3) Starting with simple delta (storing coef differences instead of original coefs),
its usually possible to gain something by predicting the next value and
subtracting the prediction before coding.

I had done some tests with simple prediction recently and wasn't successful with that, but probably that test code had some bugs or something. Will explore this further.

So while I can't say that reaching better compression by
preprocessing is impossible, but to acquire good results
you'd have to use some kind of statistical ranking model,
so its imho much more reasonable to just write a specialized
coder right away.

In fact, I had thought about this quite a lot. My motivation to create unpackmp2 in the first place is to use it for myself and to learn something while writing, testing and improving the program. I can't spend a lot of time on this (usually just a few hours during one or two nights per week) and therefore opted for the most simple and straightforward approach, which in my opinion is a small stand-alone preprocessor program. The actual compression part is slightly out of my scope at the moment.

I really appreciate the input I get from you and others here. This the main reason why I posted the program in this forum.

[Shelwien]

> But I recognise your point that for most uses it would be
> better to preserve the original mp2 with all its non-audio
> data, such as ID3 tags at head or tail of file and the
> "filler" data inside of the mp2 frames. I'm going to add
> that to the next version.

Unfortunately that's not all.
A practical recompressor also has to be able to losslessly
transform _any_ data, not only valid data in its format.
Otherwise even a mostly valid file, with a transmission
error in one byte, won't be losslessly recompressed
(or might even cause a crash).
And btw, one good reason for lossless recompression
is container format integrity. For example, suppose
we used another preprocessor to extract a mp2 audio
track from some movie, and that preprocessor is also
able to losslessly reconstruct the video container,
but only if its output is not modified.

>> It should be better to store the coefs by columns
> I think unpackmp2 already does this: it orders the scale
> factors and samples by subband and groups many consecutive
> frames in order to create longer runs of similar bytes.
> This is a kind of transforming the "2D tables" of data
> into "1D streams", isn't it?

Sure, but I have doubts about such subband serializing,
as well as zigzags suggested in the previous post.
Supposing that
...aaaabbbbbcccccc...
...AAAABBBBBCCCCCC...
turns into
...aaaaAAAAbbbbbBBBBBccccccCCCCCC...
(if that's what you meant)
the compressor would try to encode
"A" in context of last "a" coef in subband.
Or we can use zigzag here, but that
would only preserve the context for one
coef in subband.
Anyway, I tried various permutations and same
coef from previous row seemed like the best context.
Of course, my model uses other coefs as well,
but imho that can't be simulated with preprocessing+
compression with prefix contexts.

> Unpackmp2 "inflates" nearly all data elements of the mp2
> stream to full bytes. [...] It seems to help compressors
> quite a lot.

Then there's also an idea to separate the alphabets for
different data types, so their statistics won't mix.
For example, if we need to preprocess 32-bit values for
compression with a bytewise compressor, it can work
like this:
12 34 56 78 -> 71 62 53 44 35 26 17 08
This commonly increases compression despite apparent
redundancy, because otherwise the compressor would still
predict higher probability of 12 in 2nd byte of the
number, even if its a long table of numbers which all look
like 1234xxxx.

> The actual compression part is slightly out of my scope at
> the moment.

Actually context model + arithmetic coding may be simpler
than many preprocessing tricks, while providing better
compression.
I can patch up a simple demo for you, if you want.

[foxyshadis]

Unfortunately that's not all.
A practical recompressor also has to be able to losslessly
transform _any_ data, not only valid data in its format.
Otherwise even a mostly valid file, with a transmission
error in one byte, won't be losslessly recompressed
(or might even cause a crash).
And btw, one good reason for lossless recompression
is container format integrity. For example, suppose
we used another preprocessor to extract a mp2 audio
track from some movie, and that preprocessor is also
able to losslessly reconstruct the video container,
but only if its output is not modified.

I disagree. If the stream's broken, it's not generally going to be recompressible without breaking it, in the same way it's no longer correctly decodable. The only thing you could possibly do in that situation is have a marker that would indicate an uncompressed block coming up, then attempt to scan for the next sync point and start again. (Fortunately, MPEG standards have a lot of resiliency built in, even in raw streams.)

The recompressor shouldn't concern itself with meta information, sync markers, and encapsulation. If that's desired, and I agree that it is, implement it in another layer. By giving up some of the compression, you could actually stick the raw data back into MPEG frames, but it would make more sense to parse the frames and put them elsewhere (beginning or end, perhaps).

Actually context model + arithmetic coding may be simpler
than many preprocessing tricks, while providing better
compression.
I can patch up a simple demo for you, if you want.

Agreed, I don't think there's ever going to be a good way to compress AC components better than arithmetic. They are essentially somewhat random components no matter how you cut and slice them, though DC components are strongly correlated.

[Shelwien]

> I disagree. If the stream's broken, it's not generally
> going to be recompressible without breaking it, in the
> same way it's no longer correctly decodable. The only
> thing you could possibly do in that situation is have a
> marker that would indicate an uncompressed block coming
> up, then attempt to scan for the next sync point and start
> again. (Fortunately, MPEG standards have a lot of
> resiliency built in, even in raw streams.)

As I said, there're use cases where such preprocessor
is applied along with other filters - like precomp or
avi parser. And in such a case data transformations not being
completely lossless won't allow to reconstruct the
original data at all.
Of course, some patching framework can be added to
ensure that data is always losslessy reconstructed,
but that would be 2-3x slower (forward pass, backward pass,
patch generation), and not streamable.

Anyway, as an example, we have a mp3 preprocessor here,
which _does_ it like I described - for any broken or
unknown data.
http://encode.dreamhosters.com/showp...0&postcount=19

> The recompressor shouldn't concern itself with meta
> information, sync markers, and encapsulation. If that's
> desired, and I agree that it is, implement it in another
> layer.

Layers or anything... it would become much slower and awkward
if you'd implement it as a multipass algorithm, and its much
more natural to include the necessary validation into the
parser code.

[smack]

v1.2 (2010-04-02)
preserve non-audio data before first and after last mp2 frame (e.g. ID3 tags).
preserve non-audio data inside of mp2 frames.
(format of unpacked files is not compatible with previous version!)


Previous versions of the program unpacked the audio data only, while dropping all non-audio stuff from the mp2 files. That transformation could be called "lossless" only when the decoded audio content of the original and the transformed mp2 files was compared.

This new version of the program also stores non-audio data from the mp2 files. This means that original and transformed mp2 files are identical bit for bit (except for bugs...), which can be verified using tools like fc or diff.

There is no improvement in compression this time. Some tests were done to re-order the unpacked data or to employ linear prediction, but to no avail, LPAQ8 or LZMA compression could not be improved this way.