Results 1 to 17 of 17

Thread: Fast decompression of big files

  1. #1
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    873
    Thanks
    47
    Thanked 106 Times in 84 Posts
    i need to compress som big file 600mb to 4.5Gb.
    i need as god compression af possible but with fast decompression speed.
    compression speed is ok to be slow.
    comrpession requrimetns its ok to use up to 8gb of ram
    Decomrpesseion memory req. must be below 2gb


    so far i've used rep and CCM(x).
    what id did was to use several trials wit different settings of ccm(x), and just keep the fast one without much increase in file size.
    but even standard ccm has to slow decompression



    what is the best way to to get god compression ratio with fast decompression?

  2. #2
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by SvenBent
    what is the best way to to get god compression ratio with fast decompression?
    7-Zips LZMA...

  3. #3
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,474
    Thanks
    26
    Thanked 121 Times in 95 Posts
    cabarc has super fast decompression but it supports dictionaries only up to 2 mb.

    7- zip badly inflates incompressible files.

    rar with disabled ppm is somewhat good for mixed data.

  4. #4
    Member
    Join Date
    Aug 2007
    Posts
    30
    Thanks
    0
    Thanked 0 Times in 0 Posts
    lzturbo: linear scalable parallel compressor. Look at http://consultant-berater.de/lzturbo for the proper switches.

  5. #5
    Member
    Join Date
    Aug 2007
    Posts
    30
    Thanks
    0
    Thanked 0 Times in 0 Posts
    lzturbo:No other compression program decompress faster with better compression.

  6. #6
    Member
    Join Date
    Jun 2008
    Location
    G
    Posts
    372
    Thanks
    26
    Thanked 22 Times in 15 Posts
    Quote Originally Posted by SvenBent
    comrpession requrimetns its ok to use up to 8gb of ram
    yeah with try to compress it with 7z lzma

    would be interessesting how 7zip performs with a dictonary of 512mb or above

    can you poste the results here?

  7. #7
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,474
    Thanks
    26
    Thanked 121 Times in 95 Posts
    btw:
    Quote Originally Posted by SvenBent
    god compression ratio
    interesting

  8. #8
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    873
    Thanks
    47
    Thanked 106 Times in 84 Posts
    stupid me. forgot some important info.
    what kind of data it is.

    Its cd and dvd game images for my home server.

    I'm moving all my games down in the basement. and going to put the images on my server for easier access and a prettier living room (wives seems to hate 400 cd hanging on the wall )

  9. #9
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    873
    Thanks
    47
    Thanked 106 Times in 84 Posts
    lzturbo fail miserably

    TestFile.ISO - 4.26GB
    TestFile.ISO.rep - 4.22GB 2m52s
    TestFile.ISO.cmx - 3.99GB - 52m
    TestFile.ISO.lzt - 273MB - 2m02

    Do i need to tell the i wasn't able to decompress the lzturbo file

  10. #10
    Member
    Join Date
    May 2008
    Location
    Earth
    Posts
    115
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Have you tried an appropriate 64-bit 7-zip version( for intel or AMD platform?)

  11. #11
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    561
    Thanks
    212
    Thanked 199 Times in 92 Posts
    Before compressing the data, first have a look if Precomp can
    be used for preprocessing.
    Especially for games, Precomp often improves compression,
    and it has slow precompression, but very fast recompression,
    just as you need it. The -slow switch also could lead to
    further improvement if there are some non-standard zLib
    streams.
    http://schnaader.info
    Damn kids. They're all alike.

  12. #12
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    873
    Thanks
    47
    Thanked 106 Times in 84 Posts
    #scnaader thanks I will lok into it.

    so something like this:
    Image file -> ECM -> precomp -> REP -> 7zip LZMA/WinRar ?

    any good switches/settings for 7-zip/WinRar to get god compression with fast decompression ?

  13. #13
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,501
    Thanks
    741
    Thanked 664 Times in 358 Posts
    Quote Originally Posted by SvenBent
    ECM -> precomp -> REP -> 7zip LZMA/WinRar ?
    or just freearc -max actually, try RZM as the last stage

  14. #14
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    873
    Thanks
    47
    Thanked 106 Times in 84 Posts
    hehe freearc needs a gui like 7-zip right click gui.
    all the command parameters scares me..either that or I'm to lazy to learn

    RZM I'll try and find it. thanx

  15. #15
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 153 Times in 18 Posts
    Quote Originally Posted by SvenBent
    RZM Ill try and find it. thanx
    Actually, you just have to to look 2 or 3 topics up/down.

  16. #16
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,501
    Thanks
    741
    Thanked 664 Times in 358 Posts
    Quote Originally Posted by SvenBent
    hehe freearc needs a gui like 7-zip right click gui.
    http://mega-set.com/arc/eng/

    fas own gui is also on the way

  17. #17
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    873
    Thanks
    47
    Thanked 106 Times in 84 Posts
    Just back with a little decompression benchmark again

    Madagascar the game iso preprocessed with rep,

    filename - size - decomp time

    MADGSCR_1.iso.rep - 501.594.263 bytes - 0s
    MADGSCR_1.iso.rep.7z - 499.144.953 bytes - 55s
    MADGSCR_1.iso.rep.lzt - 499.028.655 bytes - 12s
    MADGSCR_1.iso.rep.rzm - 492.746.651 bytes - 130s


    i will try to post more decompression benchmarks of these 3 methods

Similar Threads

  1. convert swf files to avi files
    By Jabilo in forum The Off-Topic Lounge
    Replies: 13
    Last Post: 26th October 2016, 12:39
  2. REP and Delta fails with big files
    By SvenBent in forum Data Compression
    Replies: 14
    Last Post: 23rd November 2008, 20:41
  3. I volunteer for some big dictionary test
    By SvenBent in forum Data Compression
    Replies: 18
    Last Post: 8th June 2008, 23:59
  4. Replies: 4
    Last Post: 17th March 2008, 22:19
  5. Start of another BIG and really real Benchmark
    By Simon Berger in forum Forum Archive
    Replies: 31
    Last Post: 15th November 2007, 17:18

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •