jampack delta filter

[Shelwien]

Lucas told me about the delta filter in his jampack BWT codec (https://github.com/loxxous/Jampack/),
so I decided to test it and made a standalone version:
http://nishi.dreamhosters.com/u/lucas_v0.rar

Maybe I did something wrong? Or is it only applicable to wavs?

41,262,256 powerarc.exe
41,263,516 powerarc.luc // lucas.exe c powerarc.exe 1
 7,517,782 0.pa         // 7z.exe a -mx=9 -mf=off -bb3 -m0=plzma 0.pa powerarc.exe
 7,591,896 1_luc.pa     // 7z.exe a -mx=9 -mf=off -bb3 -m0=plzma 1_luc.pa powerarc.luc            
 7,392,726 2_deltb.pa   // 7z.exe a -mx=9 -mf=off -bb3 -m0=deltb -m1=plzma 2_deltb.pa powerarc.exe

34,943,708 black.wav             // 16bit/stereo/44100hz
34,944,776 black.luc             // lucas.exe c black.wav black.luc
34,943,716 black.mm              // mm.exe black.wav black.mm
29,852,127 3_lzma.pa             // 7z.exe a -mx=9 -mf=off -bb3 -m0=lzma 3_lzma.pa black.wav
28,327,241 4_lzma_lp2.pa         // 7z.exe a -mx=9 -mf=off -bb3 -m0=lzma:lc0:lp2:pb2 4_lzma_lp2.pa black.wav
23,853,252 5_myx9.pa             // 7z.exe a -mx=9 -myx=9 -bb3 -m0=lzma:lc0:lp2:pb2 5_myx9.pa black.wav
23,853,252 6_delta4_lzma_lp2.pa  // 7z.exe a -mx=9 -mf=off -bb3 -m0=delta:4 -m1=lzma:lc0:lp2:pb2 6_delta4_lzma_lp2.pa black.wav
23,414,405 7_luc_myx9.pa         // 7z.exe a -mx=9 -myx=9 -bb3 -m0=lzma:lc0:lp2:pb2 7_luc_myx9.pa black.luc
23,626,308 8_mm.pa               // 7z.exe a -mx=9 -mf=off -bb3 -m0=lzma:lc0:lp2:pb2 8_mm.pa black.mm

3,085,422 sample.bmp             // 
3,085,518 sample.luc             // mm.exe sample.bmp sample.mm      
3,085,431 sample.mm              // lucas.exe c sample.bmp sample.luc
1,962,465 9_myx9.pa              // 7z.exe a -mx=9 -myx=9 -bb3 9_myx9.pa sample.bmp    
1,533,116 a_luc_myx9.pa          // 7z.exe a -mx=9 -myx=9 -bb3 b_mm_myx9.pa sample.mm  
1,532,452 b_mm_myx9.pa           // 7z.exe a -mx=9 -myx=9 -bb3 a_luc_myx9.pa sample.luc

3,471,552 lzt24                  // 
3,471,658 lzt24.luc              // lucas.exe c lzt24 lzt24.luc
3,471,553 lzt24.mm               // mm.exe lzt24 lzt24.mm      
1,262,996 d_myx9.pa              // 7z.exe a -mx=9 -myx=9 -bb3 d_myx9.pa lzt24                
1,278,317 e_luc_myx9.pa          // 7z.exe a -mx=9 -myx=9 -bb3 e_luc_myx9.pa lzt24.luc        
1,262,472 f_mm_myx9.pa           // 7z.exe a -mx=9 -myx=9 -bb3 f_mm_myx9.pa lzt24.mm          
1,261,744 g_deltb_myx9.pa        // 7z.exe a -mx=9 -myx=9 -bb3 -mf=deltb g_deltb_myx9.pa lzt24

[Lucas]

Thanks for the post! From the looks of it your compile is fixed to use the brute force mode which doesn't take into account contextual entropy of deltas, the heuristic mode is the more interesting one since it runs in O(n) time and is usually more accurate than the brute force mode simply because of the "CalculateSortedEntropy" function (which is a 2N generalized bwt followed by order-1 entropy) vs "CalculateMixedEntropy" which is order-0 and order-1 averaged together.

There are no dedicated models for wav, it's just designed to be as generic as possible, it detects the width of channels using a heuristic or a brute force method, then it encodes the best channels. It's fairly strong on wav because of it's use of linear prediction as well as standard and non-standard delta. With mid-side encoding I bet I can improve compression but that would likely be overkill for a non-data specific filter.

Here's an exe with all the filter modes, multi-thread support, and variable block-sizes to play around with. And there's a small checksum to make sure it all worked. Have fun!

[Shelwien]

Point is, Bulat's delta and mm filters seem to have better results, aside from wavs -
http://web.archive.org/web/201702050.../Research.aspx

[Lucas]

The test files from https://encode.su/threads/2414-Prime-Number-Benchmark is a good example of why a non-data specific filter is useful.

prime.tar            4,812,800 bytes
prime.tar.bcm        1,108,378 bytes (no filter)
prime.tar.delta.bcm  556,401 bytes (bulat's delta filter)
prime.tar.jdelta.bcm 367,143 bytes (heuristic filter)
prime.tar.jdelta.bcm 358,783 bytes (brute filter)

The filter in Jampack is mainly designed for embedded content, like single files with a mix of wav, bmp, trimesh... etc (anything with fixed points). These are common in videogame patches and dlc.

dawnguard.bsa              1,187,540,597 (skyrim dlc)
dawnguard.bsa.bcm          650,869,137  (no filter, bcm v1.21, standard 32MB block size)
dawnguard.bsa.delta.bcm    632,299,653 (bulat's delta filter)
dawnguard.bsa.jdelta.bcm   620,667,759 (heuristic filter)
dawnguard.bsa.jdelta.bcm   619,171,343 (brute filter)

This file is an interesting one to benchmark filters. https://d3ijcvgxwtkjmf.cloudfront.ne...lw8-12.bingeom

[Bulat Ziganshin]

detection heuristics for multimedia and binary tables are just different. freearc delta is good for tables, hence it usually fails on games, increasing archive size. this one is for multimedia, so it fails on exe/database files