Results 1 to 28 of 28

Thread: Lossless (Re)compression of Ogg files?

  1. #1
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts

    Lossless (Re)compression of Ogg files?

    Ogg (vorbis) files are usually highly compressed, so compressing them further is very difficult.

    Oggre compresses ogg files fairly well, but not as much as I would like. I have heaps of ogg files that I would like to compress for archiving purposes.

    Attached sample at 1,806,612 bytes compresses down to 1,336,503 bytes using oggre. That is a ~ 26% reduction in size! Not bad.

    Tried using paq8px v183fix1, resulting in a size of 1,530,168 bytes. A 15% reduction in size. Not so good.

    But is there a way to compress the ogg file even further? Maybe by converting to some other format first? Or using some combination of (pre)compressors?

    I don't seem to be getting good results using precomp, reflate, pzlib, and/or paq8px_183fix1.
    Attached Files Attached Files

  2. #2
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    331
    Thanks
    191
    Thanked 55 Times in 39 Posts

  3. #3
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts
    Thanks for the suggestion.

    Tried ogg_rehuff and got 1,473,277 bytes. A reduction of around 18%. Better than paq8px183fix1, but still no better than oggre at 26% reduction.

  4. #4
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,842
    Thanks
    288
    Thanked 1,244 Times in 697 Posts
    1. ogg_rehuff outputs .ogg so you can combine it with oggre... There's not much gain for basement.ogg though.
    Also ogg_rehuff is lossy (huffman code optimizer; not possible to restore original ogg file) and kinda unstable - might
    be a good idea to verify it via decoding to wav in batch mode.

    2. paq needs specialized models to recompress compressed formats. It doesn't have such a model for ogg/vorbis.

    3. cdm can recompress oggs a little: https://encode.su/threads/2742-Compr...ll=1#post52493

    4. There was this old project: http://www.balatoniresearch.eu/vorbiscompressor

    So yeah, nothing better than oggre atm. At best you can try asking oggre's author to write metainfo to 2nd file (to compress it with paq or something).

    Also maybe check for jpegs (with precomp) - at least mp3s frequently contain a thumbnail jpeg which can be also recompressed.

  5. Thanks:

    brispuss (17th December 2019)

  6. #5
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts
    Did some testing by compressing 348 ogg (vorbis) files.

    Code:
    Original files size            44,053,199 bytes
    fast paq8 v6                   43,377,412          option -8
    paq8pxd v69                    42,234,615          option -s9
    oggre                          41,387,188
    Noticed that oggre compressed large ogg (vorbis) files fairly well, but conversely made worse compression on smaller files!

    Overall oggre is the best for this test, but only just. paq8pxd_v69 is close behind.

    And overall, oggre reduced files size by "only" 6%!! Not really a good result!

    Didn't test using paq8px v183fix1 because it would take a long time to compress these 348 files, and compression result most likely would only be slightly better than paq8pxd v69 result.

  7. #6
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    611
    Thanks
    246
    Thanked 240 Times in 119 Posts
    Results seem to depend heavily on the type (and content?) of the file, e.g. I found a game with ogg sounds and music that compress by about 30%. Two examples:

    Code:
    trapped.ogg	       450.551
    trapped.pcf            423.795 (LZMA2)
    trapped.paq8px         400.166 (paq8px182fix2 -8, time: 300 s)
    trapped.oggre          316.435 (oggre v0.1.1, time: 300 ms)
    
    poison-pavilion.ogg    3.536.599
    poison-pavilion.pcf    3.271.647
    poison-pavilion.ogr    2.314.530, time: 2 s
    http://schnaader.info
    Damn kids. They're all alike.

  8. Thanks:

    brispuss (20th December 2019)

  9. #7
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts
    Thanks for this test!

    Possibly the amount of compression achieved is dependent on the audio bit rate as well as on other variables? This is something I have yet to look into.

  10. #8
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    63
    Thanks
    31
    Thanked 14 Times in 11 Posts
    I've never heard of Oggre. I browsed the web and couldn't find anything about it.
    Can somebody provide a link to it, please? Is the way Oggre process Vorbis streams known?

  11. #9
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    611
    Thanks
    246
    Thanked 240 Times in 119 Posts
    Quote Originally Posted by WinnieW View Post
    I've never heard of Oggre. I browsed the web and couldn't find anything about it.
    Can somebody provide a link to it, please? Is the way Oggre process Vorbis streams known?
    Oggre v0.1.1 at Krinkels.org. As the download requires registration, I also attached a zip file containing the binaries.

    Read somewhere that it optimizes "codebooks" (one of ogg's structures). It also seems to do some deduplication according to the usage output:

    Code:
    OGGRE v0.1.1
    usage: oggre_enc.exe [-sX] [-dd] [-df] [-ds] infile outfile
           -sX  -  set books statistic buffer size, X = [0;3]. Default: -s1
           -dd  -  disable ogg deduplication
           -df  -  disable setup frame deduplication
           -ds  -  disable solid mode
    Unfortunately, it seems to be discontinued and closed source and nobody else seems to have done something similar (yet).
    Attached Files Attached Files
    http://schnaader.info
    Damn kids. They're all alike.

  12. Thanks (3):

    kaitz (21st December 2019),Shelwien (21st December 2019),WinnieW (20th December 2019)

  13. #10
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    505
    Thanks
    207
    Thanked 343 Times in 182 Posts
    Quote Originally Posted by brispuss View Post
    Did some testing by compressing 348 ogg (vorbis) files.

    Code:
    Original files size            44,053,199 bytes
    fast paq8 v6                   43,377,412          option -8
    paq8pxd v69                    42,234,615          option -s9
    oggre                          41,387,188
    Noticed that oggre compressed large ogg (vorbis) files fairly well, but conversely made worse compression on smaller files!

    Overall oggre is the best for this test, but only just. paq8pxd_v69 is close behind.

    And overall, oggre reduced files size by "only" 6%!! Not really a good result!

    Didn't test using paq8px v183fix1 because it would take a long time to compress these 348 files, and compression result most likely would only be slightly better than paq8pxd v69 result.
    Ups
    KZo


  14. #11
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    536
    Thanks
    236
    Thanked 90 Times in 70 Posts
    The format I believe is the most popular now is opus. And maybe even more in the future bc the big names are backing it. A lossless recompressor for opus would be great for android backups, for example (whatsapp folder). And an AAC recompressor for video files

  15. #12
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,842
    Thanks
    288
    Thanked 1,244 Times in 697 Posts
    Here's my old proof-of-concept AAC-LC recompressor: http://nishi.dreamhosters.com/u/aaczip_v0.rar
    As to opus, I think ogg-vorbis has higher priority (since oggre is not open-source). Unfortunately it wasn't as easy to implement as AAC when I looked - afaik
    groups of coefs were combined during quantization, so its not possible to just remove huffman and encode coefs with mp3zip model like I did with AAC.

    Although you can post an opus sample and I'd see how well cdm handles it :)

    ...Not much I guess:
    Code:
    !------------------------!--------!--------!--------!--------!--------!--------!------!
    ! Filename               !  (1)   !  (2)   !  (3)   !  (4)   !  (5)   !  (6)   ! (7)  !
    !------------------------!--------!--------!--------!--------!--------!--------!------!
    ! path.opus_vbr          !2234992 !2235022 !2233298 !2232289 !2230621*!2231432 ! 0.20%!
    ! path.opus_cbr-64k      !1499460 !1498412 !1496434 !1495231 !1485162*!1486029 ! 0.88%!
    ! ff-16b-1c-44100hz.opus !1679614 !1679626 !1678452 !1677888 !1676131*!1676859 ! 0.21%!
    ! ff-16b-2c-44100hz.opus !2598621 !2598595 !2595940 !2593616 !2593176*!2594384 ! 0.21%!
    ! gs-16b-1c-44100hz.opus ! 131566 ! 131730 ! 131498 ! 131474 ! 131432*! 131474 ! 0.10%!
    ! gs-16b-2c-44100hz.opus ! 219956 ! 220126 ! 219821 ! 219801*! 219855 ! 219881 ! 0.07%!
    !------------------------!--------!--------!--------!--------!--------!--------!------!
    ! (1) original size                                                                   !
    ! (2) 7z -mx=9 compression                                                            !
    ! (3) direct test                                                                     !
    ! (4) bytewise bwts (5M block)                                                        !
    ! (5) bit7-0 bitwise bwts (5M block)                                                  !
    ! (6) bit0-7 bitwise bwts (5M block)                                                  !
    ! (7) gain: best result vs min of (1) and (2)                                         !
    !-------------------------------------------------------------------------------------!
    Update: added samples from https://docs.espressif.com/projects/...o-samples.html

  16. #13
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    63
    Thanks
    31
    Thanked 14 Times in 11 Posts
    Both AAC and Ogg Vorbis use Huffman based prefix code as entropy codes for the bitstream.
    Opus however uses Range Coder as entropy code for the bitstream.

    Opus documentation
    https://tools.ietf.org/html/rfc6716

  17. #14
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,842
    Thanks
    288
    Thanked 1,244 Times in 697 Posts
    Yeah, but its not necessarily bad for recompression. Vorbis redundancy is not much higher: https://encode.su/threads/2742-Compr...l?p=52493&pp=1
    But with opus it might be much simpler to use strong CM, since with RC there's no need for complicated data structure to prepare for coding.
    Still, for vorbis there're at least game resources and some music albums to compress.
    But for opus there're no samples at all, aside from self-encoded ones.
    So no point?

  18. #15
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    536
    Thanks
    236
    Thanked 90 Times in 70 Posts
    opus audio files are one of the most dense I encountered so far, maybe with the exception of .cmix and company so w/o a de-compression stage, they can't be reduced.

    I wonder if there could be any exploitable inter-file common information...

  19. #16
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    536
    Thanks
    236
    Thanked 90 Times in 70 Posts
    Quote Originally Posted by Shelwien View Post
    Yeah, but its not necessarily bad for recompression. Vorbis redundancy is not much higher: https://encode.su/threads/2742-Compr...l?p=52493&pp=1
    But with opus it might be much simpler to use strong CM, since with RC there's no need for complicated data structure to prepare for coding.
    Still, for vorbis there're at least game resources and some music albums to compress.
    But for opus there're no samples at all, aside from self-encoded ones.
    So no point?

    On the contrary... Do you own an Android phone? Do you use WhatsApp? You might have hundreds MB of opus audio on it.

    This is from Wikipedia:

    Opus replaces both Vorbis and Speex for new applications, and several blind listening tests have ranked it higher-quality than any other standard audio format at any given bitrate until transparency is reached, including MP3, AAC, and HE-AAC
    (My underlining)
    ‚Äč
    So I guess it will only be gaining more popularity, at the expense of ogg and company

  20. #17
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,842
    Thanks
    288
    Thanked 1,244 Times in 697 Posts
    > they can't be reduced

    Well, cdm did compress the files I encoded a little.

    > Do you own an Android phone? Do you use WhatsApp?

    Only a tablet and no, so I don't have any opus files. Can you post a sample?

  21. #18
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    536
    Thanks
    236
    Thanked 90 Times in 70 Posts
    Yes, sure, a little bit they can be compressed.

    I could send you some maybe via PM or e-mail if you need them. I guess there must be some non-personal audio somewhere here but there are +4000 and I won't listen to them all hehe

  22. #19
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    63
    Thanks
    31
    Thanked 14 Times in 11 Posts
    There is the option to use the official Opus Encoder and create Opus test files for yourself.

    http://www.opus-codec.org/

  23. #20
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,842
    Thanks
    288
    Thanked 1,244 Times in 697 Posts
    I already did that, you can see the results above.
    But based on observation of mp3,aac,vorbis files appearing "in the wild", such files are frequently encoded inefficiently,
    and can be compressed more than ones encoded properly.
    In my results, you can partially see that in CBR vs VBR gains, but low-bitrate records of voice messages and such are
    probably even more redundant. I just don't have any to test.

  24. #21
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    63
    Thanks
    31
    Thanked 14 Times in 11 Posts
    Quote Originally Posted by Shelwien View Post
    ... low-bitrate records of voice messages and such are
    probably even more redundant. I just don't have any to test.
    Ok, here is a link to a package of audio files. The package also include some speech samples. Maybe you can encode them into opus format.
    They are compressed in flac format.

  25. #22
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,842
    Thanks
    288
    Thanked 1,244 Times in 697 Posts
    No, it doesn't work like that. Me finding options to get compressible opus files doesn't prove anything.
    While finding some created by an existing program or service does.

  26. #23
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    536
    Thanks
    236
    Thanked 90 Times in 70 Posts
    @shelwien: Did you read what I wrote?

    I can get some files to you alone, but I won't post them for the public to download

  27. #24
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,842
    Thanks
    288
    Thanked 1,244 Times in 697 Posts
    Sure, you can even test them yourself without uploading anything - http://nishi.dreamhosters.com/u/cdm_test_5.rar
    Just post the logs from running "test.bat sample.opus >>log.txt"

  28. #25
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    63
    Thanks
    31
    Thanked 14 Times in 11 Posts
    I wanted to provide some help for other people in that matter, but to be honest, I don't have much interest in better compression of Opus audio files.
    Usually I'm not interested in methods that provide less than ten percentage of data reduction, and suboptimal pareto efficiency.

  29. #26
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    536
    Thanks
    236
    Thanked 90 Times in 70 Posts
    Oh, I thought you maybe had some draft for a proper recompressor. I'm with WinnieW here, even paq* won't have any real success with opus files. They're just not compressible. Of course, cdm can maybe do 1% or 2% better than 7z but that's it. And it will be a very computationally expensive process.

  30. Thanks:

    WinnieW (25th December 2019)

  31. #27
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    63
    Thanks
    31
    Thanked 14 Times in 11 Posts
    For the purpose of gaining some insight there is the option to remove range coding from an Opus audio stream and recompress the resulting data using a CM-algorithm based program.
    Maybe there is some compression gain.

  32. #28
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,842
    Thanks
    288
    Thanked 1,244 Times in 697 Posts
    We tried that with mp3, it doesn't work: https://encode.su/threads/271-mp3dump

  33. Thanks:

    WinnieW (25th December 2019)

Similar Threads

  1. Lossless JPG Compression
    By brispuss in forum Data Compression
    Replies: 38
    Last Post: 25th December 2019, 10:36
  2. Replies: 14
    Last Post: 1st November 2019, 17:27
  3. Lossless MP3 compression question
    By JamesWasil in forum Data Compression
    Replies: 1
    Last Post: 29th December 2017, 14:57
  4. .bik no lossless compression?
    By Tidro in forum Data Compression
    Replies: 2
    Last Post: 4th September 2013, 18:39
  5. lossless data compression
    By SLS in forum Data Compression
    Replies: 21
    Last Post: 15th March 2011, 11:35

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •