Page 5 of 5 FirstFirst ... 345
Results 121 to 132 of 132

Thread: reflate - a new universal deflate recompressor

  1. #121
    Member
    Join Date
    Mar 2018
    Location
    sun
    Posts
    34
    Thanks
    19
    Thanked 16 Times in 7 Posts
    After so many years, seeing in your initial post that absolutely nobody properly thanked you back then, especially knowing this library changed the game and whole recompression benchmark, I did my part today... at least.

  2. #122
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,943
    Thanks
    291
    Thanked 1,286 Times in 728 Posts
    We just didn't have that feature then :)

  3. Thanks:

    nikkho (8th August 2019)

  4. #123
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts

    Question How to Use Reflate?

    I'm currently experimenting with several (pre)compressors on a variety of files with the intention of archiving the files.

    I've downloaded the latest version of "reflate" (version 1l2).

    The download includes six executables and two dynamic link library files, plus a sample file and (sample) batch files.

    Attempting to run any one of the four reflate*.exe executables in an elevated command window results in nothing apparently happening? No help, no syntax, nothing.

    The executables shar.exe and plzma.exe when run under the command window give syntax usage/help. But what are these two executables for? Why are they included?

    As per the thread title, how do you use "reflate" on files!? There are no instructions on how to install and on how to run "reflate".

    Thank you.
    Last edited by brispuss; 14th November 2019 at 02:58. Reason: Grammar change as post was moved to this thread

  5. #124
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,943
    Thanks
    291
    Thanked 1,286 Times in 728 Posts
    There's simply no usage text, nobody needed it before :)
    The .bat files are supposed to be usage examples instead.

    Its like this:
    reflate.exe c advcomp1z.tgz .unp .hif -- preprocess (.unp contains uncompressed data, .hif contains diff data for lossless restoring)
    reflate.exe d .unp .hif .rec -- restore

    or:
    reflate.exe c1 advcomp1z.tgz .unp .hif -- preprocess with specified target deflate level (-1 here; default is -6)
    reflate.exe d .unp .hif .rec -- restore

    reflate.exe uses dlls, supports nested processing (by adding more .hif filenames on cmdline) and some extra options (which most users won't need),
    while reflate_std.exe is standalone, supports only one .hif, but works with stdio pipes.

    reflate is also integrated in 7zdll: http://nishi.dreamhosters.com/u/7zdll_vF7.rar
    There you can use it like this: 7z a -m0=reflate -m1=lzma archive.pa files
    or: 7z.exe a -m0=reflate*4:x6666 -m1=lzma archive.pa advcomp1z.tgz

  6. Thanks:

    brispuss (12th November 2019)

  7. #125
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts
    Thanks for the prompt and fairly detailed reply!

    At the moment I understand some of what is being said.

    I've downloaded the 7zdll_vF7.rar file and installed it under Windows 7 64 bit.

    Unfortunately there are two issues!

    As a test I tried running the sample batch file test-mp3.bat. A command window appears briefly and then disappears with nothing more apparently being done!?

    I had a look at the batch file and found the call to the codec packmp3, but this codec is (apparently) not listed in the list of supported codecs (checked by running 7z.exe i). But I found three similar codecs listed instead -
    packmp3b
    packmp3c
    packmp3d

    So I tried substituting packmp3b for the non-existent "packmp3" codec and tried to run the command manually as - 7z a -m0=packmp3b 1.pa cat.mp3. But now get error message that 7-zip console has stopped working and the compression of file cat.mp3 was not completed. I tried this command several times with the same error appearing every time!?

    Then tried packmp3c and then packmp3d codecs instead (one at a time), and these codecs appeared to work. So there seems to be an issue with the packmp3b codec!?

    However! The codecs packmp3c and packmp3d both produced a file of only 3 kB from an initial size of 157 kB!!!

    Tried extracting these 3kB files back to cat.mp3 and they appeared to produce the same sized files (157 kB) as the original. BUT! There was a CRC failed error message each time on extraction (using command 7z.exe e 1.pa, or 7z.exe e 2.pa). And sometimes I got the 7-zip console stopped working error as well!

    I compared initial and extracted files and they were definitely NOT bit identical.

    So it appears there is something wrong with the mp3 compression and decompression using the codecs packmp3b, packmp3c, and packmp3d.

  8. #126
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,943
    Thanks
    291
    Thanked 1,286 Times in 728 Posts
    > So it appears there is something wrong with the mp3 compression and
    > decompression using the codecs packmp3b, packmp3c, and packmp3d.

    Yeah, mainly that they don't work like you expected.
    If you want default packmp3/mpzapi, you can get 7zdll_vF2, it works like you wanted.
    However now only packmp3c/d should be actually used
    (packmp3b still remains for backward compatibility, because a version of PA with it was released).

    packmp3c/d are actually versions of the same codec with different entropy models.
    packmp3c is a little stronger but slower, packmp3d is faster, but still better than previous ones.

    And so they have to be used like this:
    7z.exe a -m0=mp3det -m1=packmp3c 1c.pa cat.mp3
    or:
    7z.exe a -m0=mp3det -m1=packmp3c:mt4:c16M 1c4.pa cat.mp3
    (mt# = number of threads, c# = chunk size for a thread)

    see test-mp3b5.bat for the full script, since proper mp3 recompression
    apparently also involves dedup and jpeg recompression (for ID3 tags).

    Also I have this: http://nishi.dreamhosters.com/7zcmd.html

  9. Thanks:

    brispuss (12th November 2019)

  10. #127
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts
    Quote Originally Posted by Shelwien View Post
    . .
    reflate is also integrated in 7zdll: http://nishi.dreamhosters.com/u/7zdll_vF7.rar
    There you can use it like this: 7z a -m0=reflate -m1=lzma archive.pa files
    or: 7z.exe a -m0=reflate*4:x6666 -m1=lzma archive.pa advcomp1z.tgz
    OK. Something strange going on here!

    I believe I got reflate working under 7z a few days ago, but 7z now no longer works (properly), and can't get anything to compress!? Getting error: System ERROR Not implemented now!?

    Not sure if I used the original 7-zip installation with the updated files from file 7zdll_vF7.rar installed within the original directory (c:\program files\7-zip), or I might have used a newly created directory called c:\7z-mod with x64 version only files from file 7zdll_vF7.rar?

    The original (unmodified) 7-zip program under c:\program files\7-zip has been uninstalled. And the c:\7z-mod folder and files were deleted recently.

    I've re-created c:\7z-mod folder and extracted only the x64 files from file 7zdll_vF7.rar. This is the only 7-zip installation at the moment.

    Typing in 7z in a command window brings up 7z syntax etc. But trying a simple compress test such as - 7z a 0067jpg.7z 0067.jpg brings up the above error " . Not implemented".

    I can't even do basic compression let alone advanced stuff such as reflate etc.

    What is wrong here?

  11. #128
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,943
    Thanks
    291
    Thanked 1,286 Times in 728 Posts
    1. 7zdll works with a ".pa" format, there's no support for any other formats. You can't replace normal 7-zip with it.
    2. With 7zdll, "7z a 0067jpg.pa 0067.jpg" would compress the .jpg with lzma2, there's no codec optimization in console version.
    3. 7zdll does include same lepton and brunsli codecs (and jojpeg, but its much slower), but they can be used only with -ms=off (because both lepton and brunsli want to load the whole file),
    so for jpegs atm its better to use normal 7-zip with mfilter or 7zdll with jojpeg.

  12. #129
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts

    Question

    Thanks for the detailed comments.

    However, it doesn't entirely explain why I'm getting the "Not implemented" errors.

    I can now install the original 7-zip program and run it OK. But still getting errors when trying to compress files using files from 7zdll_vF7.rar.

    Original 7-zip program once again uninstalled.

    I've "installed" files from 7zdll_vF7.rar to c:\7zdll_vF7 directory. Set environment variables path to c:\7zdll_vf7\x64 since I'm running x64 version of Windows.

    Typing 7z in a command window brings up command syntax as expected. But trying to compress anything brings up errors again "Not implemented".

    Trying to run reflate on a small swf file (attached) with the command - 7z a -m0=reflate intenium.swf.wtf intenium.swf. But it doesn't work > "Not implemented"

    What is wrong here? Is the 7zdll installation wrong, somehow?
    Attached Files Attached Files

  13. #130
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,943
    Thanks
    291
    Thanked 1,286 Times in 728 Posts
    It says "Not implemented" about the archive format (.7z or .wtf).
    Archive name should either end with ".pa" or you need to specify the format explicitly with -tpa option.
    Also maybe rename 7zdll's 7z.exe to pa.exe - that would let you use both 7-zip and 7zdll at once.

  14. Thanks:

    brispuss (4th December 2019)

  15. #131
    Member
    Join Date
    Aug 2008
    Location
    NZ
    Posts
    59
    Thanks
    30
    Thanked 11 Times in 7 Posts
    Thanks. I've got the 7zdll "7z" working again.

    Followed your advice and renamed 7z.exe to pa.exe to avoid confusion with the original 7-zip program.

    For reflate, I think there may be options associated with it? What are those options please?

  16. #132
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,943
    Thanks
    291
    Thanked 1,286 Times in 728 Posts
    There're actually 3 different "codecs" based on same reflate in 7zdll:
    Code:
    -m0=reflate:x9
    -m0=reflate*4:x=9876
    -m0=reflate*7:x=1234567
    Thing is, its quite possible to encounter nested deflate... like .png in .docx in .zip, so reflate supports nested preprocessing...
    but each nesting level requires an extra output stream for diff data, and 7-zip framework doesn't support dynamic number of output streams,
    so I made 3 instances with different names instead: "reflate" has nesting depth 1, "reflate*4" = depth 4, "reflate*7" = depth 7...
    7-zip framework also has a limit at max 8 output streams per codec, and reflate*7 uses all of them.

    Unfortunately reflate doesn't yet have detection of zlib parameters... window is assumed to always be 32k (which is bad for pngs),
    and level has to be specified manually (via x parameter in this case; default is 6).
    Thus "-m0=reflate:x9" means "diff against zlib level 9", while "-m0=reflate*4:x=9876" means "level 9 for depth 0, level 8 for depth 1...".

    Its important to keep in mind that in 7zdll, reflate's "x" parameter clashes with global "x" parameter,
    so "7z a -m0=reflate ..." would use reflate/level6, while "7z a -mx=9 -m0=reflate ..." would suddently use reflate/level9,
    but its not a problem if reflate level is always specified manually like "7z a -mx=9 -m0=reflate:x6 ...".

    Reflate library has some other parameters, but they're not included in 7zdll atm.
    https://encode.su/threads/1399-refla...r?p=50858&pp=1

    Also keep in mind that actual cmdline using reflate would look something like this:
    pa a -m0=reflate:x6 -m1=lzma2:x9 -m2=lzma2:lc0:lp0:pb0 -mb00s0:1 -mb00s1:2 archive.pa *.pdf
    or even
    pa a -m0=reflate:x6 -m1=jojpeg -m2=lzma2:x9 -m3=lzma2:lc0:lp0:pb0 -m4=lzma2:lc0:lp0:pb0 -mb00s0:1 -mb00s1:4 -mb01s0:2 -mb01s1:3 archive.pa *.pdf
    since reflate's diff outputs are usually incompressible per deflate stream... but would be same for multiple instances of same stream.
    I have this converter for complex filter trees: http://nishi.dreamhosters.com/7zcmd.html

  17. Thanks (3):

    brispuss (4th December 2019),Bulat Ziganshin (6th December 2019),Simorq (13th December 2019)

Page 5 of 5 FirstFirst ... 345

Similar Threads

  1. lzma recompressor
    By Shelwien in forum Data Compression
    Replies: 33
    Last Post: 25th February 2016, 22:40
  2. DEFLATE/zlib implementations
    By GerryB in forum Data Compression
    Replies: 10
    Last Post: 7th May 2009, 17:03
  3. deflate model for paq8?
    By kaitz in forum Data Compression
    Replies: 2
    Last Post: 6th February 2009, 20:48
  4. Universal Archive Format
    By Bulat Ziganshin in forum Data Compression
    Replies: 1
    Last Post: 9th July 2008, 00:54
  5. Interesting Deflate source
    By encode in forum Forum Archive
    Replies: 10
    Last Post: 21st April 2008, 15:30

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •