Page 4 of 4 FirstFirst ... 234
Results 91 to 98 of 98

Thread: precomp - further compress already compressed files

  1. #91
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,133
    Thanks
    320
    Thanked 1,396 Times in 801 Posts
    I did it here: http://nishi.dreamhosters.com/u/7zdll_vF7.rar
    But it runs an actual precomp executable via pipes.
    And to make that executable, precomp source actually has to be patched - to make unique temp files in different threads, etc.
    (deflate parsing is disabled in my release, since I have reflate for that;
    it actually runs "powerarc-precomp64.exe -cn -d1 -t+PZGNFJSMB -zl -o%s. %s" - this string
    can be patched in 7z.dll, I guess).

    Code:
    Z:\004\7zdll_vF7\x64>7z.exe a -tpa -m0=precomp 1 A10.jpg
    
    7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
    
    Scanning the drive:
    1 file, 842468 bytes (823 KiB)
    
    Creating archive: 1.pa
    
    Items to compress: 1
    
    
    Files read from disk: 1
    Archive size: 673392 bytes (658 KiB)
    Everything is Ok

  2. Thanks:

    Bulat Ziganshin (17th December 2020)

  3. #92
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,133
    Thanks
    320
    Thanked 1,396 Times in 801 Posts
    I had an idea to try using precomp to patch watermarks in some pdf files.
    So I processed a pdf with precomp -cn, then patched the watermark string with spaces (same length),
    then restored the file. Seemed to work.
    Now I wonder how stable this is. Can this break the restored pdf?
    Maybe if I'd remove some data instead of patching it?

  4. Thanks (2):

    Mike (2nd January 2021),xinix (2nd January 2021)

  5. #93
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    630
    Thanks
    288
    Thanked 252 Times in 128 Posts
    The main content of the .pcf format using -cn are processed blocks and unprocessed blocks. For both types, some metadata is stored, most important in this case: the length is stored (vli encoded), no checksum is stored. So from a .pcf view, as long as the length is the same, you won't break anything.

    If the length changes, however, precomp will give an error in most cases when restoring because the following (un)processed blocks are unaligned and the supposed block type of processed blocks will be out of range.

    Of course, the pdf format can have all kinds of metadata, so the modifications are likely to fail on this side even if the length of strings doesn't change.
    http://schnaader.info
    Damn kids. They're all alike.

  6. #94
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,133
    Thanks
    320
    Thanked 1,396 Times in 801 Posts
    Yes, apparently pdf optimizers complain that the index doesn't match or something.
    But it still looks like precomp can be used for this.
    (attached example made by kampaster).
    Attached Files Attached Files

  7. Thanks:

    xinix (3rd January 2021)

  8. #95
    Member
    Join Date
    Feb 2021
    Location
    Brisbane
    Posts
    3
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Shelwien View Post
    I did it here: http://nishi.dreamhosters.com/u/7zdll_vF7.rar
    But it runs an actual precomp executable via pipes.
    And to make that executable, precomp source actually has to be patched - to make unique temp files in different threads, etc.
    (deflate parsing is disabled in my release, since I have reflate for that;
    it actually runs "powerarc-precomp64.exe -cn -d1 -t+PZGNFJSMB -zl -o%s. %s" - this string
    can be patched in 7z.dll, I guess).

    Code:
    Z:\004\7zdll_vF7\x64>7z.exe a -tpa -m0=precomp 1 A10.jpg
    
    7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
    
    Scanning the drive:
    1 file, 842468 bytes (823 KiB)
    
    Creating archive: 1.pa
    
    Items to compress: 1
    
    
    Files read from disk: 1
    Archive size: 673392 bytes (658 KiB)
    Everything is Ok
    Hi, is there a way to combine this precomp filter/codec with smart7z or maybe Mfilter so for certain file types the precomp codec can be used as pre process?

    Thank you

  9. #96
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    572
    Thanks
    245
    Thanked 98 Times in 77 Posts
    Quote Originally Posted by opxorj View Post
    Hi, is there a way to combine this precomp filter/codec with smart7z or maybe Mfilter so for certain file types the precomp codec can be used as pre process?

    Thank you
    Yes and no. For every file precomp can precompress, there probably will be a gain in the next step of compression, no matter what you use. Nevertheless, smat7z doesn't understand pcf format, which is a mix of actual data from the original file and reconstruction metadata, and they can be of pretty different data types. So it probably won't be of much help. What could be done is to first identify the file, then apply only the right transforms inside precomp (ie -cn -d0 -tj for a .jpg image) and then compress accordingly to the expected output format. But such a thing must be done inside smart7z or whatever you plan to use. The quick and dirty hack is to use batch scripts. In any case, it is doubtful you can accomplish anything like this. Precomp is pretty good at telling data types apart, and most modern general purpose compressors can also adapt to whatever you throw at them relatively easy. For example, lzma2 and other algorithms can tell if a chunk is incompressible and treat it accordingly.

  10. #97
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,133
    Thanks
    320
    Thanked 1,396 Times in 801 Posts
    > is there a way to combine this precomp filter/codec with smart7z or maybe Mfilter

    I didn't check, but in theory standard 7-zip plugins (ie Mfilter etc) can be used with my 7zdll version as is.
    Using my precomp wrapper with normal 7-zip is also possible, but it requires compiling it as a standalone dll (currently its a part of extended 7z.dll).

    1) This plugin implements blockwise MT: defaults are block=64MB, threads=4,
    but its possible to change from commandline using standard 7z syntax ("-m0=precomp:mt=2:c=256m" etc).

    2) It requires a modified precomp, since default version uses hardcoded tempfile names: https://github.com/schnaader/precomp...ecomp.cpp#L129
    and has no way to disable asking questions in console.

    3) Precomp is executed with cmdline "-cn -d1", but you can find it in the dll (unicode string) and patch to whatever you like.
    Attached Files Attached Files

  11. Thanks (3):

    kampaster (10th February 2021),Mike (9th February 2021),opxorj (9th February 2021)

  12. #98
    Member
    Join Date
    Feb 2021
    Location
    Brisbane
    Posts
    3
    Thanks
    1
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Shelwien View Post
    > is there a way to combine this precomp filter/codec with smart7z or maybe Mfilter


    I didn't check, but in theory standard 7-zip plugins (ie Mfilter etc) can be used with my 7zdll version as is.

    I've put the Mfilter files into 7zdll_vF7 pack, though 7z i can see the MFilter, running


    7z.exe a -mf=off -m0=MFilter -m1=LZMA2 -mb0s0:1 1.pa a10.jpg


    reports System ERROR:
    The parameter is incorrect.


    Maybe the archive format or mfilter is not supported via this way?


    Quote Originally Posted by Shelwien View Post
    > is there a way to combine this precomp filter/codec with smart7z or maybe Mfilter


    Using my precomp wrapper with normal 7-zip is also possible, but it requires compiling it as a standalone dll (currently its a part of extended 7z.dll).


    1) This plugin implements blockwise MT: defaults are block=64MB, threads=4,
    but its possible to change from commandline using standard 7z syntax ("-m0=precomp:mt=2:c=256m" etc).


    2) It requires a modified precomp, since default version uses hardcoded tempfile names: https://github.com/schnaader/precomp...ecomp.cpp#L129
    and has no way to disable asking questions in console.


    3) Precomp is executed with cmdline "-cn -d1", but you can find it in the dll (unicode string) and patch to whatever you like.

    Thanks for the work.

    I installed this new patched precomp build standalone dll for 7z into the Codecs folder, running -m0=precomp from command line works as expected.

    However when I defined this (with or without m1=LZMA2:x9:d128m:mt2 mb0s0:1 )


    :deflate: m0=precomp
    *.gz


    and try to add the .gz file to an archive via the "add to archive" gui, choose 'Smart7z' as archive format, click next,
    it will stuck on this and won't progress:


    Click image for larger version. 

Name:	2021-02-10-004123.png 
Views:	25 
Size:	13.0 KB 
ID:	8326

    As for MFilter side, I don't think it has built a rountine for processing .gz deflate type files using a defined external encoder

Page 4 of 4 FirstFirst ... 234

Similar Threads

  1. Test files to compress
    By KingAmada in forum Random Compression
    Replies: 2
    Last Post: 4th November 2019, 19:31
  2. How to expand .ff compressed files using Precomp & Fsum???
    By Manjunath in forum The Off-Topic Lounge
    Replies: 21
    Last Post: 7th September 2014, 14:47
  3. pim 2.9 compress mysql 5.1.32 x64 files
    By l1t in forum Data Compression
    Replies: 0
    Last Post: 23rd March 2009, 16:06
  4. Replies: 3
    Last Post: 10th November 2007, 22:32
  5. Replies: 12
    Last Post: 30th June 2007, 17:49

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •