precomp - further compress already compressed files

[Shelwien]

I did it here: http://nishi.dreamhosters.com/u/7zdll_vF7.rar
But it runs an actual precomp executable via pipes.
And to make that executable, precomp source actually has to be patched - to make unique temp files in different threads, etc.
(deflate parsing is disabled in my release, since I have reflate for that;
it actually runs "powerarc-precomp64.exe -cn -d1 -t+PZGNFJSMB -zl -o%s. %s" - this string
can be patched in 7z.dll, I guess).

Z:\004\7zdll_vF7\x64>7z.exe a -tpa -m0=precomp 1 A10.jpg

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21

Scanning the drive:
1 file, 842468 bytes (823 KiB)

Creating archive: 1.pa

Items to compress: 1


Files read from disk: 1
Archive size: 673392 bytes (658 KiB)
Everything is Ok

[Shelwien]

I had an idea to try using precomp to patch watermarks in some pdf files.
So I processed a pdf with precomp -cn, then patched the watermark string with spaces (same length),
then restored the file. Seemed to work.
Now I wonder how stable this is. Can this break the restored pdf?
Maybe if I'd remove some data instead of patching it?

[schnaader]

The main content of the .pcf format using -cn are processed blocks and unprocessed blocks. For both types, some metadata is stored, most important in this case: the length is stored (vli encoded), no checksum is stored. So from a .pcf view, as long as the length is the same, you won't break anything.

If the length changes, however, precomp will give an error in most cases when restoring because the following (un)processed blocks are unaligned and the supposed block type of processed blocks will be out of range.

Of course, the pdf format can have all kinds of metadata, so the modifications are likely to fail on this side even if the length of strings doesn't change.

[Shelwien]

Yes, apparently pdf optimizers complain that the index doesn't match or something.
But it still looks like precomp can be used for this.
(attached example made by kampaster).

[opxorj]

I did it here: http://nishi.dreamhosters.com/u/7zdll_vF7.rar
But it runs an actual precomp executable via pipes.
And to make that executable, precomp source actually has to be patched - to make unique temp files in different threads, etc.
(deflate parsing is disabled in my release, since I have reflate for that;
it actually runs "powerarc-precomp64.exe -cn -d1 -t+PZGNFJSMB -zl -o%s. %s" - this string
can be patched in 7z.dll, I guess).

Z:\004\7zdll_vF7\x64>7z.exe a -tpa -m0=precomp 1 A10.jpg

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21

Scanning the drive:
1 file, 842468 bytes (823 KiB)

Creating archive: 1.pa

Items to compress: 1


Files read from disk: 1
Archive size: 673392 bytes (658 KiB)
Everything is Ok

Hi, is there a way to combine this precomp filter/codec with smart7z or maybe Mfilter so for certain file types the precomp codec can be used as pre process?

Thank you

[Gonzalo]

Hi, is there a way to combine this precomp filter/codec with smart7z or maybe Mfilter so for certain file types the precomp codec can be used as pre process?

Thank you

Yes and no. For every file precomp can precompress, there probably will be a gain in the next step of compression, no matter what you use. Nevertheless, smat7z doesn't understand pcf format, which is a mix of actual data from the original file and reconstruction metadata, and they can be of pretty different data types. So it probably won't be of much help. What could be done is to first identify the file, then apply only the right transforms inside precomp (ie -cn -d0 -tj for a .jpg image) and then compress accordingly to the expected output format. But such a thing must be done inside smart7z or whatever you plan to use. The quick and dirty hack is to use batch scripts. In any case, it is doubtful you can accomplish anything like this. Precomp is pretty good at telling data types apart, and most modern general purpose compressors can also adapt to whatever you throw at them relatively easy. For example, lzma2 and other algorithms can tell if a chunk is incompressible and treat it accordingly.

[Shelwien]

> is there a way to combine this precomp filter/codec with smart7z or maybe Mfilter

I didn't check, but in theory standard 7-zip plugins (ie Mfilter etc) can be used with my 7zdll version as is.
Using my precomp wrapper with normal 7-zip is also possible, but it requires compiling it as a standalone dll (currently its a part of extended 7z.dll).

1) This plugin implements blockwise MT: defaults are block=64MB, threads=4,
but its possible to change from commandline using standard 7z syntax ("-m0=precomp:mt=2:c=256m" etc).

2) It requires a modified precomp, since default version uses hardcoded tempfile names: https://github.com/schnaader/precomp...ecomp.cpp#L129
and has no way to disable asking questions in console.

3) Precomp is executed with cmdline "-cn -d1", but you can find it in the dll (unicode string) and patch to whatever you like.

[opxorj]

> is there a way to combine this precomp filter/codec with smart7z or maybe Mfilter


I didn't check, but in theory standard 7-zip plugins (ie Mfilter etc) can be used with my 7zdll version as is.

I've put the Mfilter files into 7zdll_vF7 pack, though 7z i can see the MFilter, running


7z.exe a -mf=off -m0=MFilter -m1=LZMA2 -mb0s0:1 1.pa a10.jpg


reports System ERROR:
The parameter is incorrect.


Maybe the archive format or mfilter is not supported via this way?

> is there a way to combine this precomp filter/codec with smart7z or maybe Mfilter


Using my precomp wrapper with normal 7-zip is also possible, but it requires compiling it as a standalone dll (currently its a part of extended 7z.dll).


1) This plugin implements blockwise MT: defaults are block=64MB, threads=4,
but its possible to change from commandline using standard 7z syntax ("-m0=precomp:mt=2:c=256m" etc).


2) It requires a modified precomp, since default version uses hardcoded tempfile names: https://github.com/schnaader/precomp...ecomp.cpp#L129
and has no way to disable asking questions in console.


3) Precomp is executed with cmdline "-cn -d1", but you can find it in the dll (unicode string) and patch to whatever you like.

Thanks for the work.

I installed this new patched precomp build standalone dll for 7z into the Codecs folder, running -m0=precomp from command line works as expected.

However when I defined this (with or without m1=LZMA2:x9:d128m:mt2 mb0s0:1 )


:deflate: m0=precomp
*.gz


and try to add the .gz file to an archive via the "add to archive" gui, choose 'Smart7z' as archive format, click next,
it will stuck on this and won't progress:




As for MFilter side, I don't think it has built a rountine for processing .gz deflate type files using a defined external encoder