Page 2 of 5 FirstFirst 1234 ... LastLast
Results 31 to 60 of 132

Thread: reflate - a new universal deflate recompressor

  1. #31
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    630
    Thanks
    288
    Thanked 252 Times in 128 Posts
    Quote Originally Posted by Shelwien View Post
    schnaader, is there a way to configure precomp to detect all raw deflate streams, but skip
    the formats unsupported by reflate?
    Also, am I doing something wrong with that .iso? Why the precomp result is so much worse?
    Your run using "-t+zgn" or "-t-pfjsmb" together with "-d0" and "-intense" should be the closest to reflate. You could use "-t+zg" instead to avoid PNG multi parsing (intense mode will catch the streams inside PNGs, but act like reflate for multis), but this is just a minor difference. ZIP and gZip detection are needed because both formats don't include the zLib header and so are detected in brute mode only.

    Of course, "-t+ -d0 -brute" would be the closest to reflate, but is much too slow.

    The bad result for this file, at least for -d1, is a bug in Precomp 0.4.2 which is already fixed in my development version, where parts of streams that couldn't be restored are going into recursion, are decompressed and leave unnecessary bloating data. Luckily, this doesn't affect restoration PCF->original, which is still identical - this also is the reason why this bug "survived" for so long. Result for Precomp 0.4.3dev, "-c-", followed by SREP + 7-Zip Ultra is 318,671,286 bytes.

    For the -d0 results, those incomplete streams are handled correctly and the difference is caused by reflate's expected advantage to be able to handle those streams completely (with some .hif overhead, but still much better than only decompressing a part of the stream).
    Last edited by schnaader; 31st December 2011 at 18:22.
    http://schnaader.info
    Damn kids. They're all alike.

  2. #32
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,138
    Thanks
    320
    Thanked 1,401 Times in 803 Posts
    http://nishi.dreamhosters.com/u/reflate_v0c1.rar

    I've got two ideas for .hif compression improvement
    and one of them actually worked.
    As to other one, after implementing it, I surprisingly
    got a result similar to precomp on that .iso (322M).
    Now I probably know how to turn it into a real improvement,
    but its a bit complicated, so it would have to wait.

    Code:
    04-01-2012 14:17 v0c1
     + prev_length reset in matchfinder on token diff
    Code:
                   original   v0c .ref v0c1.ref   .ref0    .ref1
    2009_Gen~.pdf    2551611  1853328  1852822  2011876  1844149
    advcomp1.pdf      108823    75838    75840    79578    75842
    gs.pdf           1459619  1301889  1301850  1319064  1302637
    PartyLite.pdf   51341506 41273674 41054306 46280052 39433825
    SonyAR11-E.pdf   8186364  5320432  5316546  5597390  5265656
    pika_pika.png     723542   626492   625562   717569   609961
    FlashMX.pdf      4526946  2831690  2825835
    Code:
                                  /=== reflate_v0c ===\/=== reflate_v0c1 ===\
    --------------------------------------------------------------------------
    |file       |    .unp|   .raw|  .hif|    .7z|  gain|  .hif|    .7z|  gain|
    --------------------------------------------------------------------------
    |book1__7z  |  768771| 299731| 55802| 317792|-6.03%| 51211| 313133|-4.47%|
    |book1__izip|  768771| 312257|    17| 261261|16.33%|    17| 261261|16.33%|
    |book1__jar |  768771| 313576|  5684| 266953|14.87%|  4886| 266172|15.12%|
    |book1__kzip|  768771| 299437| 53912| 315982|-5.53%| 49969| 311839|-4.14%|
    |book1__pacl|  768771| 312802|   144| 261386|16.44%|   143| 261368|16.44%|
    |book1__pk  |  768771| 312490|  8485| 269825|13.65%|  8098| 269425|13.78%|
    |book1__pks |  768771| 311211| 14582| 275953|11.33%| 11844| 273165|12.23%|
    |book1__wz  |  768771| 312047|  1191| 262379|15.92%|  1105| 262355|15.92%|
    |wcc386_7z  |  536624| 303040| 30598| 305936|-0.96%| 28502| 303652|-0.20%|
    |wcc386_izip|  536624| 314952|    19| 274747|12.77%|    19| 274747|12.77%|
    |wcc386_jar |  536624| 314180|  3620| 278457|11.37%|  3166| 277863|11.56%|
    |wcc386_kzip|  536624| 302580| 26628| 301764| 0.27%| 24662| 299687| 0.96%|
    |wcc386_pacl|  536624| 314034|   124| 274788|12.50%|   124| 274778|12.50%|
    |wcc386_pk  |  536624| 313295|  8500| 283264| 9.59%|  8404| 283230| 9.60%|
    |wcc386_pks |  536624| 312781| 11689| 286612| 8.37%|  9987| 284884| 8.92%|
    |wcc386_wz  |  536624| 314914|   295| 274953|12.69%|   290| 275083|12.65%|
    --------------------------------------------------------------------------
    |           |10443160|4963327|221290|4512052| 9.09%|202427|4492642| 9.48%|
    --------------------------------------------------------------------------

  3. #33
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    474
    Thanked 175 Times in 85 Posts

    testing reflate

    I have finally had time to test reflate 0rc1 using c.bat.

    www.squeezechart.com/reflate0rc1.html
    www.squeezechart.com/reflate-pdf.html

    The results are interesting. Besides being incredibly fast, reflate beats precomp (normal mode) and stuffit on .gadget, .zip, .run, exe (wise installer), jar and .xps documents while it seems to have problems on open office documents from libreoffice + msoffice and some .pdf and .epub documents. If reflate had a special image compressor, it might beat stuffit on .png and .tif as well.

  4. #34
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,138
    Thanks
    320
    Thanked 1,401 Times in 803 Posts
    Thanks for the test, it would be interesting to repeat it with zlib parameter detection later.
    For now I'm more concerned about possible crashes and inability to restore the files.

    1. Its "0c1", not "0rc1".
    I use a branch naming scheme where numbers and letters are interleaved - same idea as "0.3.1", but more compact.

    2. .docx are compressed at level 1, while c.bat default is level 6.
    You can test different base levels by changing level= settings in the scripts.
    I suppose, level=9 might be better for some .zip and .gz files.

    3. Explicit recompression of png and jpeg files is planned, but there's still a lot of work to do
    to complete the deflate recompression library.

  5. #35
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    190
    Thanks
    8
    Thanked 64 Times in 33 Posts
    Does reflate start to freak out when it encounters optimized deflate streams?
    For instance with this book1 file (298608 bytes for the deflate stream +18 for the entire Gzip)

    How do you handle the Huffman tables?
    Attached Files Attached Files

  6. #36
    Member
    Join Date
    Dec 2009
    Location
    Netherlands
    Posts
    39
    Thanks
    3
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Shelwien View Post
    Thanks for the test, it would be interesting to repeat it with zlib parameter detection later.
    For now I'm more concerned about possible crashes and inability to restore the files.

    1. Its "0c1", not "0rc1".
    I use a branch naming scheme where numbers and letters are interleaved - same idea as "0.3.1", but more compact.

    2. .docx are compressed at level 1, while c.bat default is level 6.
    You can test different base levels by changing level= settings in the scripts.
    I suppose, level=9 might be better for some .zip and .gz files.

    3. Explicit recompression of png and jpeg files is planned, but there's still a lot of work to do
    to complete the deflate recompression library.
    Thanks for this great program! It sucks I only just found it, it's absolutely amazing! I have a question though
    Could you please implement a bruteforce mode, which tries all levels per stream and uses the level that produces the smallest .hif file? This could easily be done in parallel, making it heavily threaded. And as long as the stream isn't too big, the temporary files of each level could easily be cached to RAM, removing a potential I/O bottleneck. What do you think the potential savings are? Would it be worth it?

  7. #37
    Member
    Join Date
    Dec 2009
    Location
    Netherlands
    Posts
    39
    Thanks
    3
    Thanked 0 Times in 0 Posts
    I've got a bug to report! I've found 36 streams out of 33k+ streams I was trying to process with raw2hif.exe that don't restore to their original file. I've attached the problematic streams to this post. I was using the c6 and d6 switches during deflating and reflating. Hopefully they will help you debug your program. If you need anything else, just let me know. Thanks again for this great little program
    Attached Files Attached Files

  8. #38
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,138
    Thanks
    320
    Thanked 1,401 Times in 803 Posts
    @caveman:

    > Does reflate start to freak out when it encounters optimized deflate streams?

    Depends on what you call "freaking out" :)
    You can see the results with 7z/kzip in the previous posts - reflate would
    process any deflate streams, but when data parsings are significantly different
    from zlib, reflate has to store more recovery information.
    For now it means that some files would be expanded after recompression,
    but there's a hope that the redundancy in such cases can be further reduced
    using better (than zlib) matchfinders as a base (eg. winzip/pkzip).

    However I don't intend to directly support optimized-deflate parsers (like 7z),
    because even when source is available, its still relatively slow and has too many
    options to test within a sane amount of time.

    > For instance with this book1 file (298608 bytes for the deflate stream +18 for the entire Gzip)

    Thanks for the sample, though there's less redundancy on it than with 7z.
    I'd also appreciate more info about the method of its generation :)

    > How do you handle the Huffman tables?

    Same as other things, I guess.
    There're specialized algos for RLE diff and huffman lens diff.
    The standard zlib table is generated from unpacked data, diffed
    with the actual block's table and then diffs are compressed
    with a custom CM.

    @Mushoz:

    > Could you please implement a bruteforce mode, which tries all levels
    > per stream and uses the level that produces the smallest .hif file?

    Well, its planned, but I have to remove the last bugs first,
    and currently I'm kinda forced to work on another project.

    > This could easily be done in parallel, making it heavily threaded.

    In fact there're 4 threads in raw2hif/hif2raw as they are :)

    > And as long as the stream isn't too big, the temporary files of each
    > level could easily be cached to RAM, removing a potential I/O bottleneck.

    Well, I don't like temp files, and the idea is to eventually
    make a tool for transparent stream processing (ie stdin-to-stdout).
    Unfortunately it still requires a lot of work.

    > What do you think the potential savings are? Would it be worth it?

    Well, there's a commandline option for level setting, so its still
    possible to test all levels manually.
    Also there're tools like http://www.enigmaprotector.com/en/aboutvb.html
    which can be used to encapsulate reflate utils + unp/hif files into
    a single "archive".
    An example: http://nishi.dreamhosters.com/v/1_boxed.exe
    (produces advcomp1.unp in the current dir after running)

    > I've got a bug to report!

    Thanks, I've added it to my sample collection.
    Unfortunately its a very subtle bug related to stream interleaving,
    so I can't fix it immediately, but hopefully I'd get it sometime.

  9. #39
    Member caveman's Avatar
    Join Date
    Jul 2009
    Location
    Strasbourg, France
    Posts
    190
    Thanks
    8
    Thanked 64 Times in 33 Posts
    Quote Originally Posted by Shelwien View Post
    @caveman:

    Thanks for the sample, though there's less redundancy on it than with 7z.
    I'd also appreciate more info about the method of its generation
    It has been produced with early versions of kzip2gz and huffmix but most of the job is done by Ken Silverman's kzip, nothing really new.

  10. #40
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,138
    Thanks
    320
    Thanked 1,401 Times in 803 Posts
    A small bugfix (unfortunately previously reported bugs still remain)
    http://nishi.dreamhosters.com/u/reflate_v0c2.rar
    Code:
    04-01-2013 06:29 v0c2
     + BUG: uninitialized variable in thread_out
     + BUG: too strict EOF codelen check in rawdec_codes
    Also dlls for .raw recompression
    http://nishi.dreamhosters.com/u/reflate_0c2_dll_v0.rar

  11. #41
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,138
    Thanks
    320
    Thanked 1,401 Times in 803 Posts
    Just in case

    Code:
    Usage:
    
    rawdet input_file .out .str [prefix]
    
    reads the file, detects raw deflate streams in it (like precomp -brute),
    creates some output files:
     .out = input file with removed deflate streams
     .str = input file structure descriptor (offsets and lengths of any streams basically)
     [prefix][hex number].raw = any detected streams are moved to separate files,
       note that "prefix" is not necessarily a directory.
    
    
    rawrest .out .str output_file [prefix]
    
    restores the file from .out, .str and %s%08X.raw streams.
    
    
    raw2hif c[level=6] input.raw .unp .hif
    
    decodes the deflate data from a raw deflate stream into .unp, encodes the diff
    with specified zlib level to .hif.
    
    
    raw2hif d[level=6] .unp .hif output.raw
    
    compresses the .unp back into .raw, applies the diff info from .hif to restore
    the original stream.
    Note that "level" is not stored anywhere in the data, and thus has to be specified
    manually for "raw2hif d" the same way as for "raw2hif c" - if it doesn't match,
    restoring would be incorrect.

  12. #42
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,138
    Thanks
    320
    Thanked 1,401 Times in 803 Posts
    I found the cause of these incorrectly restored streams, where raw2hif fails to recompress them losslessy.
    Somehow I didn't notice before, but deflate actually supports two ways to encode match length = 258.
    What's especially interesting is that stream samples which I have for this "bug" actually make use of
    both encodings, and don't necessarily choose the shorter one or anything.
    Maybe its some watermark?

  13. #43
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    630
    Thanks
    288
    Thanked 252 Times in 128 Posts
    Quote Originally Posted by Shelwien View Post
    Somehow I didn't notice before, but deflate actually supports two ways to encode match length = 258.
    That's very interesting. Indeed, even the Deflate RFC seems to be wrong here, and there seems to be no RFC Errata. In chapter 3.2.5, it lists the representation of length codes by the values 257..285. Here is the interesting part of the table:

    Code:
         Extra
    Code Bits Length(s)
    ---- ---- -------
    281   5   131-162
    282   5   163-194
    283   5   195-226
    284   5   227-257
    285   0     258
    The entry for 284 is clearly wrong as 5 bits can encode the values 227-258.

    EDIT: I found this page that lists this ambiguity along with some other edge cases.
    Last edited by schnaader; 29th January 2013 at 01:46.
    http://schnaader.info
    Damn kids. They're all alike.

  14. #44
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,138
    Thanks
    320
    Thanked 1,401 Times in 803 Posts
    Thanks for the link.

    > Range of length symbol #284

    Actually it makes some sense, because in case of files with lots
    of very long matches, the dedicated maximum-match-length code is
    more efficient than #284 which can't be shorter than 6 bits.
    And on other hand, if len=258 is rare enough, it makes sense
    to not allocate a code for #285.

    But its not how its used where I encountered it.

    > Code length symbol #16 at beginning of sequence

    This line in my decoder:
    Code:
    if( (index==0) && (count==16) ) return -5;
    is inherited from zlib, so its an error.

    > More code lengths than needed
    > Number of literal codes
    Code:
    if( (nlen>286) || (ndist>30) ) return -3;
    > All zero code lengths for distance code
    > Encodable lengths without distance code

    Afaik zlib allows it, though maybe I should add such checks to rawdet.

    > No code for end-of-block symbol
    Code:
    if( lengths[256]==0 ) return -9;
    > Only one symbol in literal/length code or code length code

    Its ok.

    > Underfull Huffman code

    That actually happens in the length table quite frequently, even with zlib.
    I have an extra check in the huffman decoder due to that - still need to
    optimize it somehow.

  15. #45
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,138
    Thanks
    320
    Thanked 1,401 Times in 803 Posts
    http://nishi.dreamhosters.com/u/reflate_v0c3.rar
    http://nishi.dreamhosters.com/u/reflate_0c3_dll_v0.rar
    Code:
    30-01-2013 00:40 v0c3
     + BUG: lossless handling of match_len=258
    Reflate still has some problems with very small streams with weird structure like this:
    Code:
    last=0 type=1 size=9
    last=0 type=1 size=0
    last=0 type=1 size=10
    last=0 type=1 size=0
    last=0 type=1 size=19
    last=0 type=1 size=0
    last=0 type=1 size=9
    last=0 type=1 size=0
    last=0 type=1 size=8
    last=0 type=1 size=0
    last=1 type=1 size=0
    (<1k files in bug.zip from http://encode.su/threads/1399-reflat...ll=1#post30790)

    But unfortunately that's a bug in the actual diffing algo, so it might be harder to fix.
    That is, I can actually "fix" this by increasing the size of block header stack, but
    the actual bug won't disappear probably.

  16. #46
    Member
    Join Date
    Jan 2014
    Location
    Spain
    Posts
    34
    Thanks
    10
    Thanked 7 Times in 7 Posts
    Could anyone put an easy to understand example to know how to use this tool? I have experience with Precomp, but I still don't know how to use Reflate, even with the examples and batch files included in the distribution archive.

    Also, is it possible to integrate Reflate with FreeArc by modifying arc.ini? If so, how?

    Thank you very much.

  17. #47
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    343
    Thanks
    201
    Thanked 58 Times in 42 Posts
    Quote Originally Posted by hiddenuser View Post
    Could anyone put an easy to understand example to know how to use this tool? I have experience with Precomp, but I still don't know how to use Reflate, even with the examples and batch files included in the distribution archive.Also, is it possible to integrate Reflate with FreeArc by modifying arc.ini? If so, how?Thank you very much.
    The easy way is to open a command prompt and type "c.bat file.ext" (without quotes).

  18. Thanks:

    hiddenuser (1st July 2014)

  19. #48
    Member
    Join Date
    Jan 2014
    Location
    Spain
    Posts
    34
    Thanks
    10
    Thanked 7 Times in 7 Posts
    Thank you, and how can I integrate Reflate into FreeArc? Is it possible?

  20. #49
    Member
    Join Date
    May 2013
    Location
    ARGENTINA
    Posts
    54
    Thanks
    62
    Thanked 13 Times in 10 Posts
    Quote Originally Posted by hiddenuser View Post
    Thank you, and how can I integrate Reflate into FreeArc? Is it possible?
    integrate? http://www.mediafire.com/download/d4...arc_GOZARCK.7z just drag the folder to compress.

  21. Thanks:

    RamiroCruzo (9th September 2015)

  22. #50
    Member
    Join Date
    Jan 2014
    Location
    Spain
    Posts
    34
    Thanks
    10
    Thanked 7 Times in 7 Posts
    Quote Originally Posted by GOZARCK View Post
    integrate? http://www.mediafire.com/download/d4...arc_GOZARCK.7z just drag the folder to compress.
    Thank you for your efforts, but I'd like to know how to modify the arc.ini file in order to integrate Reflate without external files. I don't want to lose the customizations of my FreeArc installation.

  23. #51
    Member
    Join Date
    Oct 2014
    Location
    South Africa
    Posts
    38
    Thanks
    23
    Thanked 7 Times in 5 Posts
    Quote Originally Posted by hiddenuser View Post
    Thank you for your efforts, but I'd like to know how to modify the arc.ini file in order to integrate Reflate without external files. I don't want to lose the customizations of my FreeArc installation.
    I edited the .bat files and then make them as .exe to work with FreeArc.

    I had to use nz.exe (nanozip) for storing the decompressed file as I couldn't make it by using "arc.exe".

    Extract the attached files in "bin" folder. Then add following code to "arc.ini". The .bat files are included for modifications.

    You should combine the "reflate" with your favorite compressor to compress the data.

    Code:
    [External compressor:reflate]
    packcmd   = reflate_c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp.nz
    unpackcmd = reflate_d $$arcpackedfile$$.tmp.nz  $$arcdatafile$$.tmp
    datafile   = $$arcdatafile$$.tmp
    packedfile = $$arcpackedfile$$.tmp.nz
    Attached Files Attached Files
    Last edited by msat59; 9th November 2014 at 03:33.

  24. Thanks (2):

    hiddenuser (17th November 2014),RamiroCruzo (5th September 2015)

  25. #52
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,138
    Thanks
    320
    Thanked 1,401 Times in 803 Posts
    By request, here's an IC17 x64 build of reflate utils:
    http://nishi.dreamhosters.com/u/reflate_0c3_x64.rar
    (There's no fix for >4G deflate streams though, only executables compiled from the same old source)

    Update:
    Also the dlls: http://nishi.dreamhosters.com/u/refl...x64_dll_v0.rar

  26. Thanks (9):

    Edison007 (5th June 2016),GOZARCK (5th June 2016),kassane (13th June 2016),Mike (5th June 2016),Minimum (5th June 2016),RamiroCruzo (5th June 2016),Razor12911 (5th June 2016),schnaader (29th August 2016),xinix (5th June 2016)

  27. #53
    Member
    Join Date
    May 2013
    Location
    ARGENTINA
    Posts
    54
    Thanks
    62
    Thanked 13 Times in 10 Posts
    Big thanks Shelwien! Im so happy with that.

  28. Thanks (2):

    RamiroCruzo (5th June 2016),Stephan Busch (5th June 2016)

  29. #54
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,138
    Thanks
    320
    Thanked 1,401 Times in 803 Posts
    http://nishi.dreamhosters.com/u/7zdll_v6.rar

    Please test, there could be bugs.
    reflate:x=# sets the usual "level" parameter, still no autodetect, default is 9 (just reflate, without :x=#)
    Simple "7z a -m0=reflate:x5 -m1=plzma4 1b book1x3.gz" is also possible, but somehow results are a little worse than with explicit stream binding.

  30. Thanks (7):

    Bulat Ziganshin (29th August 2016),GOZARCK (30th August 2016),Mike (29th August 2016),Minimum (2nd September 2016),RamiroCruzo (30th August 2016),Stephan Busch (29th August 2016),xinix (30th August 2016)

  31. #55
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    474
    Thanked 175 Times in 85 Posts
    Using Reflate:x9 with LZMA2 does not seem to recompress the .run file in the Installer testset.

  32. #56
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,138
    Thanks
    320
    Thanked 1,401 Times in 803 Posts
    What about http://nishi.dreamhosters.com/u/rawfilt_v1d.rar ?
    Maybe some of the params help there?

    Update: Seems like you're doing something wrong?
    Code:
    // wget "--referer=http://www2.ati.com/drivers/linux/" http://www2.ati.com/drivers/linux/amd-driver-installer-12-2-x86.x86_64.run
    107,213,014 amd-driver-installer-12-2-x86.x86_64.run
    105,240,934 4b.7z // 7z a -m0=lzma2 4b.7z amd-driver-installer-12-2-x86.x86_64.run
     75,249,385 4a.7z // 7z a -m0=reflate:x9 -m1=lzma2 -m2=copy -mb0s0:1 -mb0s1:2 4a.7z amd-driver-installer-12-2-x86.x86_64.run
     67,832,464 4b.7z // 7z a -m0=reflate:x9 -m1=copy -m2=bcj2 -m3=lzma2 -m4=LZMA:d25 -m5=LZMA:d19 -m6=LZMA2:d19 -mb0s0:2 -mb0s1:1 -mb2s0:3 -mb2s1:4 -mb2s2:5 -mb2s3:6 4c.7z amd-driver-installer-12-2-x86.x86_64.run
    Update2: Added bcj2

  33. #57
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    474
    Thanked 175 Times in 85 Posts
    yep.. I did something wrong.. sorry..

  34. #58
    Member
    Join Date
    Jan 2016
    Location
    India
    Posts
    20
    Thanks
    23
    Thanked 8 Times in 7 Posts
    can't restore a 595,921,053 bytes file that re compressed to 945,458,905

    fc output

    Code:
    Comparing files characters.vpp_pc and .UNP
    043E790A: 38 00
    09C6F2EA: 38 00
    18290A20: 70 00
    19E43954: 38 00
    used ‚Äčhttp://nishi.dreamhosters.com/u/rawfilt_v1d.rar

    EDIT: reflate-7z module is been able to restore things back correctly
    Last edited by PrinceGupta; 2nd September 2016 at 20:42.

  35. Thanks:

    Shelwien (3rd September 2016)

  36. #59
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    4,138
    Thanks
    320
    Thanked 1,401 Times in 803 Posts
    1. Is it possible that you used different level settings for reflate.exe and 7z/reflate?
    Like, maybe the default -9 for reflate.exe and x5 from my posted cmdline for 7z?
    Then 7z with x9 might have the same bug actually...

    2. Can you tell me where to get the file?

  37. #60
    Member
    Join Date
    Jan 2016
    Location
    India
    Posts
    20
    Thanks
    23
    Thanked 8 Times in 7 Posts
    1. Is it possible that you used different level settings for reflate.exe and 7z/reflate?
    Like, maybe the default -9 for reflate.exe and x5 from my posted cmdline for 7z?
    Then 7z with x9 might have the same bug actually...
    here is the cmd line used

    Code:
    7z a -m0=reflate:x5 -m1=plzma4:a1:mt2 -m2=copy -mb0s0:1 -mb0s1:2 1a.7z characters.vpp_pc
    reflate c5 characters.vpp_pc .out .raw 1 1 1 0
    i even try
    Code:
    7z a -m0=reflate:x9 -m1=plzma4:a1:mt2 -m2=copy -mb0s0:1 -mb0s1:2 1a.7z characters.vpp_pc
    and file is still restored correctly

    2. Can you tell me where to get the file?
    The file is taken from Saint Row: 4 Actually there is a whole sets of such files which canbe only recompressed with reflate about double of their sizes bt only files<500mb can be restored correctly
    if u wish i can upload individual file on server else u can download the game there is many file that can't be restored and a lot for testing
    one more thing probably evry game of the Saint Row Series Actually have such files

Page 2 of 5 FirstFirst 1234 ... LastLast

Similar Threads

  1. lzma recompressor
    By Shelwien in forum Data Compression
    Replies: 33
    Last Post: 25th February 2016, 23:40
  2. DEFLATE/zlib implementations
    By GerryB in forum Data Compression
    Replies: 10
    Last Post: 7th May 2009, 18:03
  3. deflate model for paq8?
    By kaitz in forum Data Compression
    Replies: 2
    Last Post: 6th February 2009, 21:48
  4. Universal Archive Format
    By Bulat Ziganshin in forum Data Compression
    Replies: 1
    Last Post: 9th July 2008, 01:54
  5. Interesting Deflate source
    By encode in forum Forum Archive
    Replies: 10
    Last Post: 21st April 2008, 16:30

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •