Results 1 to 15 of 15

Thread: REP and Delta fails with big files

  1. #1
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts

    REP and Delta fails with big files

    i just tried to use REP and Delta filters on a 6.44GB file.
    and both ended up giving me a file slightly below 30kbytes

    is there a fixed version that supports big files ?

  2. #2
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    10x, will look into this

  3. #3
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by SvenBent View Post
    i just tried to use REP and Delta filters on a 6.44GB file.
    and both ended up giving me a file slightly below 30kbytes

    is there a fixed version that supports big files ?
    I tested FA -delta+rep on a 23 GB file, but could not reproduce the error you got.
    Did you use FA or the stand-alone delta.exe/rep.exe ?
    What FA version did you use ?
    Code:
    arc a -mdelta+rep:200mb test *.tib
    FreeArc 0.50 alpha (August 3 2008) started: 0.00 secs
    Compressing 1 file of 23.779.817.984 bytes: 0.01 secs
      Using delta+rep:200mb
      Memory for compression 272mb, decompression 99.9%
      Solid block compression results (592.726 seconds)
        delta: 23.783.048.828 bytes in 244.656 seconds
        rep:200mb: 22.125.748.340 bytes in 348.069 seconds
    
      Writing directory: 1237.54 secs
      Found 1 directory names: 1237.62 secs
      Directory written: 1237.6
    Compressed 1 file, 23.779.817.984 => 22.125.748.340 bytes. Ratio 93.0%
    Compression time 592.73 secs, speed 40.119 kB/s. Total 1237.63 secs

  4. #4
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    these are different things. probably my frontend used in STANDALOBE rep.delta just use 32-bit ints for filesizes. it should be easy to fix

  5. #5
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    these are different things. probably my frontend used in STANDALOBE rep.delta just use 32-bit ints for filesizes. it should be easy to fix
    I knew there are some issues with the stand-alone versions, that's the reason I don't use them only very seldom.
    Dict.exe has a problem with mixed text/binaries (tar balls) and outputs something like a 4 byte file. While you are at these, could you please also have a look at this ?
    Thanks !

  6. #6
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts
    Just to clarify.
    I use the rep /delta stand alone programs


    however i just tried with nanozip 0.0.4a and it gave me a 333kbytes file

    7-zip and winrar works perfectly
    Last edited by SvenBent; 29th September 2008 at 21:27.

  7. #7
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by SvenBent View Post
    Just to clarify.
    I use the rep /delta stand alone programs.
    Why don't you use ARC for that ? Less typing and probably faster because delta/rep can run parallel (MTT)
    however i just tried with nanozip 0.0.4a and it gave me a 333kbytes file
    That's weird : I used NZ -cD to compress my 27 GB file and it checked out ok.
    What options did you use ?

  8. #8
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts
    Quote Originally Posted by pat357 View Post
    Why don't you use ARC for that ? Less typing and probably faster because delta/rep can run parallel (MTT)

    That's weird : I used NZ -cD to compress my 27 GB file and it checked out ok.
    What options did you use ?
    i use rep/delta stand alone because i brute force combinations with rzm 7-zip and rar.


    i just used nz a -co -m1g archieve.nz file1 file2

    but could it be because that i use a network mapped drive ?

  9. #9
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    good news: i almost finished improving tornado cmdline driver and hopefully next week will release rep/delta with it. this means linux compatibility, progress indicator, large files support and lot of other facilities. also it will be BSD-licensed so i hope that other programmers will employ it too

  10. #10
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by Bulat Ziganshin View Post
    good news: i almost finished improving tornado cmdline driver and hopefully next week will release rep/delta with it. this means linux compatibility, progress indicator, large files support and lot of other facilities. also it will be BSD-licensed so i hope that other programmers will employ it too
    Well, that's really good news!

    Best regards!

  11. #11
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    i've added new cmdline driver to REP, please test it:
    Attached Files Attached Files

  12. #12
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts
    Yay new toys to test

    are there any other improvements than big file size support?
    iIremember something about the memory usage was unnecessary big(2x the size)?

  13. #13
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    870
    Thanks
    47
    Thanked 105 Times in 83 Posts
    seems, compression is a little but worse in default mode
    however -a99 had improved compression

    but i have only teste one file so far. in -b1024m and -b1024m -a99 mode
    md5 checksum after manual decompression was ok

    however i notices som chances

    -b now needs an m at the end to work EGE -b1024m instead of -b1024

    when ypu decomrpess and specify outputfile name it crashes.

    i will see i can reproduce this an nail it down to something specific

  14. #14
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    > compression is a little but worse

    yes, it's because output block is now limited to 8mb (it was up to dictsize before)

    >-b now needs an m at the end to work EGE -b1024m instead of -b1024

    yes, i think that all options that deals with sizes should use this format (with k/m/g at the end). previously i used megabytes to simplify implementation

    >when ypu decomrpess and specify outputfile name it crashes.

    probably you don't mastered new cmdline?

    rep file.rep -ooutfile

  15. #15
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    Quote Originally Posted by SvenBent View Post
    Yay new toys to test

    are there any other improvements than big file size support?
    I remember something about the memory usage was unnecessary big(2x the size)?
    yes, it's the only real improvement (and sideback that it decreased compression a tiny little)

Similar Threads

  1. FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)
    By Bulat Ziganshin in forum Data Compression
    Replies: 554
    Last Post: 26th September 2018, 02:41
  2. convert swf files to avi files
    By Jabilo in forum The Off-Topic Lounge
    Replies: 13
    Last Post: 26th October 2016, 11:39
  3. Noob question about dictionary size (and about rep)
    By SvenBent in forum Data Compression
    Replies: 1
    Last Post: 23rd January 2009, 00:35
  4. Replies: 4
    Last Post: 17th March 2008, 21:19
  5. Fast decompression of big files
    By SvenBent in forum Forum Archive
    Replies: 16
    Last Post: 8th March 2008, 19:17

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •