Looking for standalone preprocessor for testing

[chunick]

I'm looking for a standalone preprocessor for testing my compressor - something that will transform the data to be more compressible; ie. for text etc.. I'm positive I've seen one here a couple of years ago, I just want one for quickly testing final results on top of my core compression. I'll end up focusing on that portion later, I need it purely for quick and dirty use without polluting my code at the moment.

Thanks!

[inikep]

You can try this:
https://github.com/inikep/XWRT/releases

[chunick]

Perfect it has a preprocessed and uncompressed output option!

[SvenBent]

bulat made delta which is a delta filtering preprocessor

https://encode.su/threads/1045-Delta...s-preprocessor
Links are dead though

[Mauro Vezzosi]

bulat made delta which is a delta filtering preprocessor

https://encode.su/threads/1045-Delta...s-preprocessor
Links are dead though

Take a look here http://freearc.org/Research.aspx in "DELTA compression preprocessor", links work and newer version.

[Bulat Ziganshin]

not sure what type of preprocessors you need, but now all my stuff is collected in the http://freearc.org/download/research/fazip03.zip - including exe, dispack070, mm, delta, and dict preprocessors

f.e. "fazip delta <infile >outfile"

[chunick]

very helpful thanks all! ... I was just looking to test my compressor with the benefits of different preprocessors, nothing particular but dict preprocessor is nice. I'm doing a lot of fine tuning and adjustments at the moment on my core code. I don't wan't to integrate the preprocessing stage at this point but would like to see if I get a significant boost as I'm adjusting - instead of just the raw data.

Thanks!

[Bulat Ziganshin]

and yeah, i forgot the rep filter. if you need parameters, this is a list of compression methods used in fa'next m1d..m9d modes:

Compress with rep:512mb:c512+exe+4x4:zstd:2:8mb, $obj => rep:512mb:c512+4x4:zstd:2:8mb, $text => 4x4:zstd:2:8mb, $compressed => rep:512mb:c512

Compress with rep:512mb:256:c256+exe+delta+4x4:zstd:7:16m:h8m, $obj => rep:512mb:256:c256+delta+4x4:zstd:7:16m:h8m, $text => delta+4x4:zstd:7:16m:h8m, $compressed => rep:512mb:256:c256+4x4:zstd:2:8mb, $wav => tta:m1, $bmp => mm:d1+4x4:zstd:2:8mb:t0

Compress with rep:512mb+exe+delta+4x4:lzma:96mb:fast:32:mc4, $obj => rep:512mb+delta+4x4:lzma:96mb:fast:32:mc4, $text => dict:64mb:80%+4x4:lzma:96mb:fast:32:mc4:32m, $compressed => rep:512mb+4x4:zstd:2:8mb, $wav => tta:m1, $bmp => mm+4x4:lzma:96mb:fast:32:mc4

Compress with rep:512mb+dispack070+delta+4x4:lzma:96mb:normal:16 :mc8, $obj => rep:512mb+delta+4x4:lzma:96mb:normal:16:mc8, $text => dict:64mb:75%+4x4:lzma:96mb:normal:16:mc8:32m, $compressed => rep:512mb+4x4:zstd:7:16m:h8m, $wav => tta:m1, $bmp => mm+4x4:lzma:96mb:normal:16:mc8

Compress with rep:512mb+dispack070+delta+4x4:lzma:96mb:normal:12 8:mc32, $obj => rep:512mb+delta+4x4:lzma:96mb:normal:128:mc32, $text => dict:128mb:75%+4x4:lzma:96mb:normal:128:mc32, $wav => tta:m1, $bmp => mm+4x4:lzma:96mb:normal:128:mc32

Compress with rep:2040mb+dispack070+delta+lzma:254mb:normal:bt4: 128, $obj => rep:2040mb+delta+lzma:254mb:normal:bt4:128, $text => dict:128mb:75%+lzma:254mb:normal:bt4:128, $wav => tta:m1, $bmp => mm+lzma:254mb:normal:bt4:128

each "$group=method" describes compression method for some type of files. it seems that freearc/fazip implements all filters used in modern archivers in comination with LZ-based compressors, so you will not miss any