Results 1 to 7 of 7

Thread: about rawdet

  1. #1
    Member
    Join Date
    Mar 2020
    Location
    china
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts

    about rawdet

    Hello, I used rawdet to extract gzip rdeflate streams, but found that rawdet cannot support hundreds of MB, or several gigabytes of gzip,
    ​ because the extracted deflate streams are incomplete.
    Is there any solution?

  2. #2
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,962
    Thanks
    295
    Thanked 1,293 Times in 734 Posts
    Here I made a new version using rawfilt library.
    Tell me if you need parameters (entropy threshold etc) or restore - currently it just prints 64-bit begin/end positions and dumps streams to files.
    This problem happened for unpacked size >4g-32k, due to a 32-bit unpacked size variable.
    I actually kept it intentionally, because raw2hif also has various 32-bit variables, so this misdetection bug just disabled
    recompression after a few 100s of megabytes, rather than letting raw2hif crash :)

    Code:
    Z:\007>rawdet book1x3.gz
    beg=00000000 last=0 type=2 size=32767 unplen=135228
    end=0004DB16 r=0
    beg=0004DB16 last=0 type=2 size=32767 unplen=135228
    end=0009B62C r=0
    beg=0009B62C last=0 type=2 size=32767 unplen=135228
    end=000E9142 r=0
    
    Z:\007>raw2unp 00000000.raw 00000000.unp
    
    Z:\007>md5sum book1 00000000.unp
    0a0fdbaf0589c9713bde9120cbb20199 *book1
    0a0fdbaf0589c9713bde9120cbb20199 *00000000.unp
    Attached Files Attached Files

  3. #3
    Member
    Join Date
    Mar 2020
    Location
    china
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thank you very much, hello, I can use the latest rawdet to extract the original size of the deflation, but the raw2unp decompression deflation is only 4GB in size, does raw2unp not support large files?

  4. #4
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,962
    Thanks
    295
    Thanked 1,293 Times in 734 Posts
    > I can use the latest rawdet to extract the original size of the deflation,
    > but the raw2unp decompression deflation is only 4GB in size, does raw2unp not support large files?

    x64 wasn't quite mainstream back in 2011 when I originally made reflate,
    so all the parts have similar problems, unfortunately.

    I recompiled raw2unp and added to previous post, please test.

  5. #5
    Member
    Join Date
    Mar 2020
    Location
    china
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    ​Hello, I have tested and already support large files, thank you very much. In addition, if I remove the 10,000 bytes of the gzip file header,
    and use rawdet to detect this bad gzip, he can't find the deflation, is this normal?

  6. #6
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,962
    Thanks
    295
    Thanked 1,293 Times in 734 Posts
    > if I remove the 10,000 bytes of the gzip file header, and use rawdet to detect this bad gzip, he can't find the deflation, is this normal?

    1) rawdet only looks for byte-aligned deflate blocks, while any 2nd+ deflate block can start from any bit in a byte.
    2) deflate matches would depend on 32kb of previous data, so rawdet would discard a non-first block as undecodable even if it happens to be byte-aligned.

    Some workarounds are possible and handling this is in my plans for reflate2, but who knows if it happens and when.

  7. #7
    Member
    Join Date
    Mar 2020
    Location
    china
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Okay thank you. I was testing a gzip with the header removed, but found that the rawdet could not be extracted.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •