Page 1 of 7 123 ... LastLast
Results 1 to 30 of 209

Thread: LZHAM

  1. #1
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts

    LZHAM

    I just stumbled upon a codec that does something I've been thinking to do:

    LZHAM (LZ, Huffman, Arithmetic, Markov) is a data compression library that borrows many ideas from LZMA but purposely makes several key tradeoffs that favor decompression speed over compression ratio. LZHAM's compression ratio is approximately 1.5% less than LZMA, but decompresses 3-4x faster on a Core i7.
    Sadly, I can't test it, there are 2 executables, one tries to import some kernel32.dll function which doesn't exist in XP and the other just shows something like "Hello I'm a memory->memory compressor. Tested, everything's OK." Yeah, right.

    https://code.google.com/p/lzham/
    Last edited by m^2; 17th August 2010 at 12:18.

  2. Thanks:

    WayneD (7th March 2015)

  3. #2
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by m^2 View Post
    I just stumbled upon a codec that does something I've been thinking to do:
    Sadly, I can't test it, there are 2 executables, one tries to import some kernel32.dll function which doesn't exist in XP and the other just shows something like "Hello I'm a memory->memory compressor. Tested, everything's OK." Yeah, right.
    Is there some special reason that you didn't provide a link?

    Best regards!

  4. #3
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    you forget the link: http://code.google.com/p/lzham/

  5. #4
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    As Bulat guessed, I forgot. ^^
    Updated the 1st post.

  6. #5
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Compiled it to work on XP.
    Usage:
    Code:
    lzhamtest_x64 c input_file compressed_file
    lzhamtest_x64 d compressed_file decompressed_file
    Bookstar:
    Code:
    codec        mode      size     decompression time (CPU)
    LZHAM 0x1001 uber      10893224 0.906 s.
    7za 4.63     mx=9      10744976 1.656 s.
    Arc 0.666    tor:10    11972555 1.390 s.
    Arc 0.666    tor:10:c3 11999793 1.093 s.
    Update: added Tornado to the mix. mode 10 was stronger than 11.
    Attached Files Attached Files
    Last edited by m^2; 17th August 2010 at 18:14.

  7. #6
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by m^2 View Post
    Compiled it to work on XP.
    Usage:
    Code:
    lzhamtest_x64 c input_file compressed_file
    lzhamtest_x64 d compressed_file decompressed_file
    Bookstar:
    Code:
    codec        mode size     decompression time (CPU)
    LZHAM 0x1001 uber 10893224 0.906 s.
    7za 4.63     mx=9 10744976 1.656 s.
    Thank you!

    Best regards!

  8. #7
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    I made some quick tests. Since LZHAM is a single-file compressor I tar-ed my DoubleCommander-folder first and compared LZHAM to 7z.
    LZHAM:
    Code:
    Microsoft Windows XP [Version 5.1.2600]
    (C) Copyright 1985-2001 Microsoft Corp.
    
    I:\Tmp\lzham>lzhamtest_x86_default c K:\win7PE\Programs\doublecmd.tar test.lzham
    
    Dynamically loading DLL "I:\Tmp\lzham\lzham_x86.dll"
    Loaded LZHAM DLL version 0x1001
    Testing: Streaming compression
    [...]
    Success
    Input file size: 44443136, Compressed file size: 11830544, Ratio: 73.38%
    Compression time: 274.63
    Consumption rate:  161832.1 bytes/sec, Emission rate:   43078.9 bytes/sec
    Input file adler32: 0xD1B99793
    7-Zip:
    Code:
    I:\Tmp\lzham>timer 7za a test.lzma K:\win7PE\Programs\doublecmd.tar
    Timer 9.01 : Igor Pavlov : Public domain : 2009-05-31
    
    7-Zip (A) 9.15 beta  Copyright (c) 1999-2010 Igor Pavlov  2010-06-20
    Scanning
    
    Creating archive test.lzma
    
    Compressing  doublecmd.tar
    
    Everything is Ok
    
    Kernel Time  =     0.578 =    0%
    User Time    =    65.593 =   97%
    Process Time =    66.171 =   98%
    Global Time  =    66.981 =  100%
    Where file-size is -> 11.524.366 Bytes
    Unpacking
    LZHAM:
    Code:
    I:\Tmp\lzham>lzhamtest_x86_default d test.lzham test
    Dynamically loading DLL "I:\Tmp\lzham\lzham_x86.dll"
    Loaded LZHAM DLL version 0x1001
    Testing: Streaming decompression
    Success
    Compressed file size: 11830552, Decompressed file size: 44443136
    Decompression time: 3.27
    Consumption rate: 3623446.2 bytes/sec, Decompression rate: 13611986.5 bytes/sec
    Decompressed adler32: 0xD1B99793
    7-Zip
    Code:
    I:\Tmp\lzham>I:\Tmp\lzham>timer 7za x test.lzma
    Timer 9.01 : Igor Pavlov : Public domain : 2009-05-31
    
    7-Zip (A) 9.15 beta  Copyright (c) 1999-2010 Igor Pavlov  2010-06-20
    
    Processing archive: test.lzma
    
    Extracting  doublecmd.tar
    
    Everything is Ok
    
    Size:       44443136
    Compressed: 11524366
    
    Kernel Time  =     0.375 =    8%
    User Time    =     2.078 =   46%
    Process Time =     2.453 =   54%
    Global Time  =     4.500 =  100%
    
    I:\Tmp\lzham>
    Best regards!
    Last edited by Vacon; 17th August 2010 at 16:58. Reason: added file-size of 7-Zip's lzma

  9. #8
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    215
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Some more tests:

    Intel T7200 detuned to 1.00 GHz (for cooling purposes)
    test.tar, 283 MB (Mathcad 14 distribution)
    No code has to be inserted here.
    Last edited by nanoflooder; 17th August 2010 at 18:42. Reason: table ftw

  10. #9
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Another file, installer (the one I used for testing century ago):
    No code has to be inserted here.

    Also, I updated the previous post with Tor results.

  11. #10
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Bulat, what do you think about adding LZHAM to FreeArc at some point?

  12. #11
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    i think that such compressor is useful but wait for better solution

  13. #12
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    So you don't find it as amazing as I do?

  14. #13
    Member Fu Siyuan's Avatar
    Join Date
    Apr 2009
    Location
    Mountain View, CA, US
    Posts
    176
    Thanks
    10
    Thanked 17 Times in 2 Posts
    Quote Originally Posted by m^2 View Post
    So you don't find it as amazing as I do?
    I think the shining point of LZHAM is it creatively achives high ratio by Huffman-like coder.While LZX is obsoleted and LZMA is LZ+Ari which decompresses slower.
    However, I think it isn't matured enough, should have potential to be improved. There should be no reason that LZHAM is slower in compression than LZMA since LZHAM use similar ideas.

  15. #14
    Member Fu Siyuan's Avatar
    Join Date
    Apr 2009
    Location
    Mountain View, CA, US
    Posts
    176
    Thanks
    10
    Thanked 17 Times in 2 Posts
    http://www.ezcodesample.com/

    I found this site very practical and interesting, it's the reference mentioned in LZHAM site.

  16. #15
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    Tested. http://mattmahoney.net/dc/text.html#2246

    I found the idea of Polar codes interesting. There is a simple algorithm to calculate them and the compression is only slightly worse than Huffman codes.

  17. #16
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Matt, can you test different LZHAM modes too?
    You can find the execs attached here.

  18. #17
    Member zody's Avatar
    Join Date
    Aug 2009
    Location
    Germany
    Posts
    90
    Thanks
    0
    Thanked 1 Time in 1 Post
    New Alpha2 is here.. click me

  19. #18
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    XP compatible compiles, also with modes other than default.
    ADDED:
    Added modes that use more memory.
    Attached Files Attached Files
    Last edited by m^2; 27th August 2010 at 18:29.

  20. #19
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    TCUP:
    No code has to be inserted here.
    Exe:
    No code has to be inserted here.Quick tests. Seems that compression is unchanged

  21. #20
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    lzham alpha 3.6 test program supports all of the library options. The max options improve compression without much speed penalty. http://mattmahoney.net/dc/text.html#2140

  22. #21
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Matt Mahoney View Post
    lzham alpha 3.6 test program supports all of the library options. The max options improve compression without much speed penalty. http://mattmahoney.net/dc/text.html#2140
    lzham deserves an underscore now.
    Nothing stronger decompresses faster.
    Idea for LTCB:
    How about treating memory needed for compression and decompression independently?

  23. #22
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by m^2 View Post
    lzham deserves an underscore now. [...]
    well, it's on it's way.
    I found some things to mention:
    - the alpha 3 runs on XP out-of-the-box (so m^2 is un-employed now )
    - if I run "lzhamtest_x86 a test-folder" a temporary file is made that vanishes after compression, but when I give a name to create the archive -> "lzhamtest_x86 a test-folder test.lzham" it complains "Error: too many filenames!". Any hints? Am I too tired to see the right command?

    Best regards!

  24. #23
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    Yes, you're right. lzham is on the Pareto frontier for decompression time. Probably should list memory separately for decompression but don't feel like retesting everything.

  25. #24
    Member
    Join Date
    Jul 2006
    Location
    US
    Posts
    39
    Thanks
    26
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Matt Mahoney View Post
    lzham alpha 3.6 test program [...] http://mattmahoney.net/dc/text.html#2140
    Matt, a minor point.

    On LTCB you refer to "lzham alpha 2.7". Looking at the official LZHAM pages, I don't see any such version.

    Maybe you confused the ".7" from the beginning of the 7zip file extension of the distribution archive ("lzham_alpha2.7z") as being part of the version number?

    Initially, I thought that your reference to "alpha 3.6" was also an error/typo but you could be referring to a build of alpha3 revision6 sources, yeah?

  26. #25
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    lzham - a rising sun with the capabilities of the 7z - archivfile - format ?


    @vacon: "Am I too tired to see the right command?"

    for me it seems

    the switch "a" = Recursively compress all files under "inpath" is only for test purposes

    there is also not a switch "x" = extract inclusively folder

    @m^2: "so m^2 is un-employed now"

    can you fix this problem ? (not deleting the resulting file)

    best regards

  27. #26
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by joerg View Post
    @m^2: "so m^2 is un-employed now"

    can you fix this problem ? (not deleting the resulting file)

    best regards
    Sorry, no. The XP fix was removal 4 bytes from one file in pretty obvious places. Fixing a regular bug requires some understanding and this is less critical and I'm not willing to do it now.

  28. #27
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    The LZHAM's author run LTCB with 512 MB dictionary.
    Click.

  29. #28
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    You're right about the version numbers. I included the author's test results. http://mattmahoney.net/dc/text.html#2065

  30. #29
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    I have now tried several times to file an issue for lzham (see above -> "a"-switch), but everytime I submit it I get "400"-error:
    Bad Request
    Your client has issued a malformed or illegal request.
    Has anyone else tried to get in contact with the author?

    Best regards!

  31. #30
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Alpha 4, supposedly stronger and faster, released.

Page 1 of 7 123 ... LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •