Results 1 to 8 of 8

Thread: Looking for standalone preprocessor for testing

  1. #1
    Member
    Join Date
    Feb 2013
    Location
    Canada
    Posts
    4
    Thanks
    3
    Thanked 0 Times in 0 Posts

    Looking for standalone preprocessor for testing

    I'm looking for a standalone preprocessor for testing my compressor - something that will transform the data to be more compressible; ie. for text etc.. I'm positive I've seen one here a couple of years ago, I just want one for quickly testing final results on top of my core compression. I'll end up focusing on that portion later, I need it purely for quick and dirty use without polluting my code at the moment.

    Thanks!

  2. #2
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts

  3. #3
    Member
    Join Date
    Feb 2013
    Location
    Canada
    Posts
    4
    Thanks
    3
    Thanked 0 Times in 0 Posts
    Perfect it has a preprocessed and uncompressed output option!

  4. #4
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    919
    Thanks
    57
    Thanked 113 Times in 90 Posts
    bulat made delta which is a delta filtering preprocessor

    https://encode.su/threads/1045-Delta...s-preprocessor
    Links are dead though

  5. Thanks:

    chunick (9th February 2017)

  6. #5
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    273
    Thanks
    114
    Thanked 159 Times in 116 Posts
    Quote Originally Posted by SvenBent View Post
    bulat made delta which is a delta filtering preprocessor

    https://encode.su/threads/1045-Delta...s-preprocessor
    Links are dead though
    Take a look here http://freearc.org/Research.aspx in "DELTA compression preprocessor", links work and newer version.

  7. Thanks:

    chunick (9th February 2017)

  8. #6
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,564
    Thanks
    775
    Thanked 687 Times in 372 Posts
    not sure what type of preprocessors you need, but now all my stuff is collected in the http://freearc.org/download/research/fazip03.zip - including exe, dispack070, mm, delta, and dict preprocessors

    f.e. "fazip delta <infile >outfile"

  9. #7
    Member
    Join Date
    Feb 2013
    Location
    Canada
    Posts
    4
    Thanks
    3
    Thanked 0 Times in 0 Posts
    very helpful thanks all! ... I was just looking to test my compressor with the benefits of different preprocessors, nothing particular but dict preprocessor is nice. I'm doing a lot of fine tuning and adjustments at the moment on my core code. I don't wan't to integrate the preprocessing stage at this point but would like to see if I get a significant boost as I'm adjusting - instead of just the raw data.

    Thanks!

  10. #8
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,564
    Thanks
    775
    Thanked 687 Times in 372 Posts
    and yeah, i forgot the rep filter. if you need parameters, this is a list of compression methods used in fa'next m1d..m9d modes:

    Compress with rep:512mb:c512+exe+4x4:zstd:2:8mb, $obj => rep:512mb:c512+4x4:zstd:2:8mb, $text => 4x4:zstd:2:8mb, $compressed => rep:512mb:c512

    Compress with rep:512mb:256:c256+exe+delta+4x4:zstd:7:16m:h8m, $obj => rep:512mb:256:c256+delta+4x4:zstd:7:16m:h8m, $text => delta+4x4:zstd:7:16m:h8m, $compressed => rep:512mb:256:c256+4x4:zstd:2:8mb, $wav => tta:m1, $bmp => mm:d1+4x4:zstd:2:8mb:t0

    Compress with rep:512mb+exe+delta+4x4:lzma:96mb:fast:32:mc4, $obj => rep:512mb+delta+4x4:lzma:96mb:fast:32:mc4, $text => dict:64mb:80%+4x4:lzma:96mb:fast:32:mc4:32m, $compressed => rep:512mb+4x4:zstd:2:8mb, $wav => tta:m1, $bmp => mm+4x4:lzma:96mb:fast:32:mc4

    Compress with rep:512mb+dispack070+delta+4x4:lzma:96mb:normal:16 :mc8, $obj => rep:512mb+delta+4x4:lzma:96mb:normal:16:mc8, $text => dict:64mb:75%+4x4:lzma:96mb:normal:16:mc8:32m, $compressed => rep:512mb+4x4:zstd:7:16m:h8m, $wav => tta:m1, $bmp => mm+4x4:lzma:96mb:normal:16:mc8

    Compress with rep:512mb+dispack070+delta+4x4:lzma:96mb:normal:12 8:mc32, $obj => rep:512mb+delta+4x4:lzma:96mb:normal:128:mc32, $text => dict:128mb:75%+4x4:lzma:96mb:normal:128:mc32, $wav => tta:m1, $bmp => mm+4x4:lzma:96mb:normal:128:mc32

    Compress with rep:2040mb+dispack070+delta+lzma:254mb:normal:bt4: 128, $obj => rep:2040mb+delta+lzma:254mb:normal:bt4:128, $text => dict:128mb:75%+lzma:254mb:normal:bt4:128, $wav => tta:m1, $bmp => mm+lzma:254mb:normal:bt4:128
    each "$group=method" describes compression method for some type of files. it seems that freearc/fazip implements all filters used in modern archivers in comination with LZ-based compressors, so you will not miss any
    Last edited by Bulat Ziganshin; 10th February 2017 at 04:54.

Similar Threads

  1. Testing for open-sources libraries on POWER8
    By nemequ in forum Data Compression
    Replies: 0
    Last Post: 2nd September 2016, 22:37
  2. Fuzz testing
    By m^2 in forum Data Compression
    Replies: 23
    Last Post: 10th January 2016, 09:01
  3. Dict preprocessor
    By pat357 in forum Data Compression
    Replies: 5
    Last Post: 2nd May 2014, 21:51
  4. Testing compressors with artificial data
    By Matt Mahoney in forum Data Compression
    Replies: 0
    Last Post: 25th April 2013, 08:22
  5. What's wrong with my testing script?
    By m^2 in forum Data Compression
    Replies: 20
    Last Post: 21st September 2008, 19:24

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •