Results 1 to 5 of 5

Thread: jampack delta filter

  1. #1
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,977
    Thanks
    296
    Thanked 1,305 Times in 741 Posts

    jampack delta filter

    Lucas told me about the delta filter in his jampack BWT codec (https://github.com/loxxous/Jampack/),
    so I decided to test it and made a standalone version:
    http://nishi.dreamhosters.com/u/lucas_v0.rar

    Maybe I did something wrong? Or is it only applicable to wavs?
    Code:
    41,262,256 powerarc.exe
    41,263,516 powerarc.luc // lucas.exe c powerarc.exe 1
     7,517,782 0.pa         // 7z.exe a -mx=9 -mf=off -bb3 -m0=plzma 0.pa powerarc.exe
     7,591,896 1_luc.pa     // 7z.exe a -mx=9 -mf=off -bb3 -m0=plzma 1_luc.pa powerarc.luc            
     7,392,726 2_deltb.pa   // 7z.exe a -mx=9 -mf=off -bb3 -m0=deltb -m1=plzma 2_deltb.pa powerarc.exe
    
    34,943,708 black.wav             // 16bit/stereo/44100hz
    34,944,776 black.luc             // lucas.exe c black.wav black.luc
    34,943,716 black.mm              // mm.exe black.wav black.mm
    29,852,127 3_lzma.pa             // 7z.exe a -mx=9 -mf=off -bb3 -m0=lzma 3_lzma.pa black.wav
    28,327,241 4_lzma_lp2.pa         // 7z.exe a -mx=9 -mf=off -bb3 -m0=lzma:lc0:lp2:pb2 4_lzma_lp2.pa black.wav
    23,853,252 5_myx9.pa             // 7z.exe a -mx=9 -myx=9 -bb3 -m0=lzma:lc0:lp2:pb2 5_myx9.pa black.wav
    23,853,252 6_delta4_lzma_lp2.pa  // 7z.exe a -mx=9 -mf=off -bb3 -m0=delta:4 -m1=lzma:lc0:lp2:pb2 6_delta4_lzma_lp2.pa black.wav
    23,414,405 7_luc_myx9.pa         // 7z.exe a -mx=9 -myx=9 -bb3 -m0=lzma:lc0:lp2:pb2 7_luc_myx9.pa black.luc
    23,626,308 8_mm.pa               // 7z.exe a -mx=9 -mf=off -bb3 -m0=lzma:lc0:lp2:pb2 8_mm.pa black.mm
    
    3,085,422 sample.bmp             // 
    3,085,518 sample.luc             // mm.exe sample.bmp sample.mm      
    3,085,431 sample.mm              // lucas.exe c sample.bmp sample.luc
    1,962,465 9_myx9.pa              // 7z.exe a -mx=9 -myx=9 -bb3 9_myx9.pa sample.bmp    
    1,533,116 a_luc_myx9.pa          // 7z.exe a -mx=9 -myx=9 -bb3 b_mm_myx9.pa sample.mm  
    1,532,452 b_mm_myx9.pa           // 7z.exe a -mx=9 -myx=9 -bb3 a_luc_myx9.pa sample.luc
    
    3,471,552 lzt24                  // 
    3,471,658 lzt24.luc              // lucas.exe c lzt24 lzt24.luc
    3,471,553 lzt24.mm               // mm.exe lzt24 lzt24.mm      
    1,262,996 d_myx9.pa              // 7z.exe a -mx=9 -myx=9 -bb3 d_myx9.pa lzt24                
    1,278,317 e_luc_myx9.pa          // 7z.exe a -mx=9 -myx=9 -bb3 e_luc_myx9.pa lzt24.luc        
    1,262,472 f_mm_myx9.pa           // 7z.exe a -mx=9 -myx=9 -bb3 f_mm_myx9.pa lzt24.mm          
    1,261,744 g_deltb_myx9.pa        // 7z.exe a -mx=9 -myx=9 -bb3 -mf=deltb g_deltb_myx9.pa lzt24

  2. Thanks (2):

    spwolf (21st September 2017),xinix (22nd September 2017)

  3. #2
    Member
    Join Date
    Feb 2015
    Location
    United Kingdom
    Posts
    176
    Thanks
    28
    Thanked 74 Times in 44 Posts
    Thanks for the post! From the looks of it your compile is fixed to use the brute force mode which doesn't take into account contextual entropy of deltas, the heuristic mode is the more interesting one since it runs in O(n) time and is usually more accurate than the brute force mode simply because of the "CalculateSortedEntropy" function (which is a 2N generalized bwt followed by order-1 entropy) vs "CalculateMixedEntropy" which is order-0 and order-1 averaged together.

    There are no dedicated models for wav, it's just designed to be as generic as possible, it detects the width of channels using a heuristic or a brute force method, then it encodes the best channels. It's fairly strong on wav because of it's use of linear prediction as well as standard and non-standard delta. With mid-side encoding I bet I can improve compression but that would likely be overkill for a non-data specific filter.

    Here's an exe with all the filter modes, multi-thread support, and variable block-sizes to play around with. And there's a small checksum to make sure it all worked. Have fun!
    Attached Files Attached Files

  4. Thanks (2):

    Shelwien (22nd September 2017),spwolf (22nd September 2017)

  5. #3
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,977
    Thanks
    296
    Thanked 1,305 Times in 741 Posts
    Point is, Bulat's delta and mm filters seem to have better results, aside from wavs -
    http://web.archive.org/web/201702050.../Research.aspx

  6. #4
    Member
    Join Date
    Feb 2015
    Location
    United Kingdom
    Posts
    176
    Thanks
    28
    Thanked 74 Times in 44 Posts
    The test files from https://encode.su/threads/2414-Prime-Number-Benchmark is a good example of why a non-data specific filter is useful.
    Code:
    prime.tar            4,812,800 bytes
    prime.tar.bcm        1,108,378 bytes (no filter)
    prime.tar.delta.bcm  556,401 bytes (bulat's delta filter)
    prime.tar.jdelta.bcm 367,143 bytes (heuristic filter)
    prime.tar.jdelta.bcm 358,783 bytes (brute filter)
    The filter in Jampack is mainly designed for embedded content, like single files with a mix of wav, bmp, trimesh... etc (anything with fixed points). These are common in videogame patches and dlc.
    Code:
    dawnguard.bsa              1,187,540,597 (skyrim dlc)
    dawnguard.bsa.bcm          650,869,137  (no filter, bcm v1.21, standard 32MB block size)
    dawnguard.bsa.delta.bcm    632,299,653 (bulat's delta filter)
    dawnguard.bsa.jdelta.bcm   620,667,759 (heuristic filter)
    dawnguard.bsa.jdelta.bcm   619,171,343 (brute filter)
    This file is an interesting one to benchmark filters. https://d3ijcvgxwtkjmf.cloudfront.ne...lw8-12.bingeom
    Attached Files Attached Files

  7. #5
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,572
    Thanks
    780
    Thanked 687 Times in 372 Posts
    detection heuristics for multimedia and binary tables are just different. freearc delta is good for tables, hence it usually fails on games, increasing archive size. this one is for multimedia, so it fails on exe/database files

Similar Threads

  1. Jampack
    By Lucas in forum Data Compression
    Replies: 11
    Last Post: 27th June 2017, 01:10
  2. Genomic filter
    By Marco_B in forum Data Compression
    Replies: 25
    Last Post: 14th December 2015, 11:52
  3. EOL filter
    By kaitz in forum Data Compression
    Replies: 0
    Last Post: 29th September 2014, 23:43
  4. Simple WAV Delta Filter
    By david_werecat in forum Data Compression
    Replies: 5
    Last Post: 6th February 2012, 00:17
  5. About filter
    By vcore in forum Forum Archive
    Replies: 4
    Last Post: 22nd January 2008, 12:45

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •