Lossless (Re)compression of Ogg files?

[brispuss]

Ogg (vorbis) files are usually highly compressed, so compressing them further is very difficult.

Oggre compresses ogg files fairly well, but not as much as I would like. I have heaps of ogg files that I would like to compress for archiving purposes.

Attached sample at 1,806,612 bytes compresses down to 1,336,503 bytes using oggre. That is a ~ 26% reduction in size! Not bad.

Tried using paq8px v183fix1, resulting in a size of 1,530,168 bytes. A 15% reduction in size. Not so good.

But is there a way to compress the ogg file even further? Maybe by converting to some other format first? Or using some combination of (pre)compressors?

I don't seem to be getting good results using precomp, reflate, pzlib, and/or paq8px_183fix1.

[comp1]

Have you tried this?
http://nishi.dreamhosters.com/u/ogg_rehuff.rar

[brispuss]

Thanks for the suggestion.

Tried ogg_rehuff and got 1,473,277 bytes. A reduction of around 18%. Better than paq8px183fix1, but still no better than oggre at 26% reduction.

[Shelwien]

1. ogg_rehuff outputs .ogg so you can combine it with oggre... There's not much gain for basement.ogg though.
Also ogg_rehuff is lossy (huffman code optimizer; not possible to restore original ogg file) and kinda unstable - might
be a good idea to verify it via decoding to wav in batch mode.

2. paq needs specialized models to recompress compressed formats. It doesn't have such a model for ogg/vorbis.

3. cdm can recompress oggs a little: https://encode.su/threads/2742-Compr...ll=1#post52493

4. There was this old project: http://www.balatoniresearch.eu/vorbiscompressor

So yeah, nothing better than oggre atm. At best you can try asking oggre's author to write metainfo to 2nd file (to compress it with paq or something).

Also maybe check for jpegs (with precomp) - at least mp3s frequently contain a thumbnail jpeg which can be also recompressed.

[brispuss]

Did some testing by compressing 348 ogg (vorbis) files.

Original files size            44,053,199 bytes
fast paq8 v6                   43,377,412          option -8
paq8pxd v69                    42,234,615          option -s9
oggre                          41,387,188

Noticed that oggre compressed large ogg (vorbis) files fairly well, but conversely made worse compression on smaller files!

Overall oggre is the best for this test, but only just. paq8pxd_v69 is close behind.

And overall, oggre reduced files size by "only" 6%!! Not really a good result!

Didn't test using paq8px v183fix1 because it would take a long time to compress these 348 files, and compression result most likely would only be slightly better than paq8pxd v69 result.

[schnaader]

Results seem to depend heavily on the type (and content?) of the file, e.g. I found a game with ogg sounds and music that compress by about 30%. Two examples:

trapped.ogg	       450.551
trapped.pcf            423.795 (LZMA2)
trapped.paq8px         400.166 (paq8px182fix2 -8, time: 300 s)
trapped.oggre          316.435 (oggre v0.1.1, time: 300 ms)

poison-pavilion.ogg    3.536.599
poison-pavilion.pcf    3.271.647
poison-pavilion.ogr    2.314.530, time: 2 s

[brispuss]

Thanks for this test!

Possibly the amount of compression achieved is dependent on the audio bit rate as well as on other variables? This is something I have yet to look into.

[WinnieW]

I've never heard of Oggre. I browsed the web and couldn't find anything about it.
Can somebody provide a link to it, please? Is the way Oggre process Vorbis streams known?

[schnaader]

I've never heard of Oggre. I browsed the web and couldn't find anything about it.
Can somebody provide a link to it, please? Is the way Oggre process Vorbis streams known?

Oggre v0.1.1 at Krinkels.org. As the download requires registration, I also attached a zip file containing the binaries.

Read somewhere that it optimizes "codebooks" (one of ogg's structures). It also seems to do some deduplication according to the usage output:

OGGRE v0.1.1
usage: oggre_enc.exe [-sX] [-dd] [-df] [-ds] infile outfile
       -sX  -  set books statistic buffer size, X = [0;3]. Default: -s1
       -dd  -  disable ogg deduplication
       -df  -  disable setup frame deduplication
       -ds  -  disable solid mode

Unfortunately, it seems to be discontinued and closed source and nobody else seems to have done something similar (yet).

[kaitz]

Did some testing by compressing 348 ogg (vorbis) files.

Original files size            44,053,199 bytes
fast paq8 v6                   43,377,412          option -8
paq8pxd v69                    42,234,615          option -s9
oggre                          41,387,188

Noticed that oggre compressed large ogg (vorbis) files fairly well, but conversely made worse compression on smaller files!

Overall oggre is the best for this test, but only just. paq8pxd_v69 is close behind.

And overall, oggre reduced files size by "only" 6%!! Not really a good result!

Didn't test using paq8px v183fix1 because it would take a long time to compress these 348 files, and compression result most likely would only be slightly better than paq8pxd v69 result.

Ups

[Gonzalo]

The format I believe is the most popular now is opus. And maybe even more in the future bc the big names are backing it. A lossless recompressor for opus would be great for android backups, for example (whatsapp folder). And an AAC recompressor for video files

[Shelwien]

Here's my old proof-of-concept AAC-LC recompressor: http://nishi.dreamhosters.com/u/aaczip_v0.rar
As to opus, I think ogg-vorbis has higher priority (since oggre is not open-source). Unfortunately it wasn't as easy to implement as AAC when I looked - afaik
groups of coefs were combined during quantization, so its not possible to just remove huffman and encode coefs with mp3zip model like I did with AAC.

Although you can post an opus sample and I'd see how well cdm handles it :)

...Not much I guess:

!------------------------!--------!--------!--------!--------!--------!--------!------!
! Filename               !  (1)   !  (2)   !  (3)   !  (4)   !  (5)   !  (6)   ! (7)  !
!------------------------!--------!--------!--------!--------!--------!--------!------!
! path.opus_vbr          !2234992 !2235022 !2233298 !2232289 !2230621*!2231432 ! 0.20%!
! path.opus_cbr-64k      !1499460 !1498412 !1496434 !1495231 !1485162*!1486029 ! 0.88%!
! ff-16b-1c-44100hz.opus !1679614 !1679626 !1678452 !1677888 !1676131*!1676859 ! 0.21%!
! ff-16b-2c-44100hz.opus !2598621 !2598595 !2595940 !2593616 !2593176*!2594384 ! 0.21%!
! gs-16b-1c-44100hz.opus ! 131566 ! 131730 ! 131498 ! 131474 ! 131432*! 131474 ! 0.10%!
! gs-16b-2c-44100hz.opus ! 219956 ! 220126 ! 219821 ! 219801*! 219855 ! 219881 ! 0.07%!
!------------------------!--------!--------!--------!--------!--------!--------!------!
! (1) original size                                                                   !
! (2) 7z -mx=9 compression                                                            !
! (3) direct test                                                                     !
! (4) bytewise bwts (5M block)                                                        !
! (5) bit7-0 bitwise bwts (5M block)                                                  !
! (6) bit0-7 bitwise bwts (5M block)                                                  !
! (7) gain: best result vs min of (1) and (2)                                         !
!-------------------------------------------------------------------------------------!

Update: added samples from https://docs.espressif.com/projects/...o-samples.html

[WinnieW]

Both AAC and Ogg Vorbis use Huffman based prefix code as entropy codes for the bitstream.
Opus however uses Range Coder as entropy code for the bitstream.

Opus documentation
https://tools.ietf.org/html/rfc6716

[Shelwien]

Yeah, but its not necessarily bad for recompression. Vorbis redundancy is not much higher: https://encode.su/threads/2742-Compr...l?p=52493&pp=1
But with opus it might be much simpler to use strong CM, since with RC there's no need for complicated data structure to prepare for coding.
Still, for vorbis there're at least game resources and some music albums to compress.
But for opus there're no samples at all, aside from self-encoded ones.
So no point?

[Gonzalo]

opus audio files are one of the most dense I encountered so far, maybe with the exception of .cmix and company so w/o a de-compression stage, they can't be reduced.

I wonder if there could be any exploitable inter-file common information...

[Gonzalo]

Yeah, but its not necessarily bad for recompression. Vorbis redundancy is not much higher: https://encode.su/threads/2742-Compr...l?p=52493&pp=1
But with opus it might be much simpler to use strong CM, since with RC there's no need for complicated data structure to prepare for coding.
Still, for vorbis there're at least game resources and some music albums to compress.
But for opus there're no samples at all, aside from self-encoded ones.
So no point?

On the contrary... Do you own an Android phone? Do you use WhatsApp? You might have hundreds MB of opus audio on it.

This is from Wikipedia:

Opus replaces both Vorbis and Speex for new applications, and several blind listening tests have ranked it higher-quality than any other standard audio format at any given bitrate until transparency is reached, including MP3, AAC, and HE-AAC

(My underlining)
​
So I guess it will only be gaining more popularity, at the expense of ogg and company

[Shelwien]

> they can't be reduced

Well, cdm did compress the files I encoded a little.

> Do you own an Android phone? Do you use WhatsApp?

Only a tablet and no, so I don't have any opus files. Can you post a sample?

[Gonzalo]

Yes, sure, a little bit they can be compressed.

I could send you some maybe via PM or e-mail if you need them. I guess there must be some non-personal audio somewhere here but there are +4000 and I won't listen to them all hehe

[WinnieW]

There is the option to use the official Opus Encoder and create Opus test files for yourself.

http://www.opus-codec.org/

[Shelwien]

I already did that, you can see the results above.
But based on observation of mp3,aac,vorbis files appearing "in the wild", such files are frequently encoded inefficiently,
and can be compressed more than ones encoded properly.
In my results, you can partially see that in CBR vs VBR gains, but low-bitrate records of voice messages and such are
probably even more redundant. I just don't have any to test.

[WinnieW]

... low-bitrate records of voice messages and such are
probably even more redundant. I just don't have any to test.

Ok, here is a link to a package of audio files. The package also include some speech samples. Maybe you can encode them into opus format.
They are compressed in flac format.

[Shelwien]

No, it doesn't work like that. Me finding options to get compressible opus files doesn't prove anything.
While finding some created by an existing program or service does.

[Gonzalo]

@shelwien: Did you read what I wrote?

I can get some files to you alone, but I won't post them for the public to download

[Shelwien]

Sure, you can even test them yourself without uploading anything - http://nishi.dreamhosters.com/u/cdm_test_5.rar
Just post the logs from running "test.bat sample.opus >>log.txt"

[WinnieW]

I wanted to provide some help for other people in that matter, but to be honest, I don't have much interest in better compression of Opus audio files.
Usually I'm not interested in methods that provide less than ten percentage of data reduction, and suboptimal pareto efficiency.

[Gonzalo]

Oh, I thought you maybe had some draft for a proper recompressor. I'm with WinnieW here, even paq* won't have any real success with opus files. They're just not compressible. Of course, cdm can maybe do 1% or 2% better than 7z but that's it. And it will be a very computationally expensive process.

[WinnieW]

For the purpose of gaining some insight there is the option to remove range coding from an Opus audio stream and recompress the resulting data using a CM-algorithm based program.
Maybe there is some compression gain.

[Shelwien]

We tried that with mp3, it doesn't work: https://encode.su/threads/271-mp3dump