Results 1 to 26 of 26

Thread: Fast, portable, open source LZH?

  1. #1
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts

    Fast, portable, open source LZH?

    I'm looking for compressors to include in my fsbench...and have problems. These don't have to be LZH, but shouldn't have decompression speed much worse than deflate and should offer fair compression speed / strength on small data.

    So far I found just 1 suitable candidate, it's zlib.

    I added Tornado too and I intend to add LZHAM alpha 8 when it's out because the author promised much improved small block performance, but these aren't great.
    I *may* add 7-zip's deflate and deflate64, but I'm not eager to do it because I don't like having to use 7-zip on Windows and p7zip on *nix...

    Do I miss something?

    Why are there so few players in this category?

  2. #2
    Member Karhunen's Avatar
    Join Date
    Dec 2011
    Location
    USA
    Posts
    91
    Thanks
    2
    Thanked 1 Time in 1 Post
    How about bzip2 http://www.bzip.org/1.0.6/bzip2-1.0.6.tar.gz and LZO http://www.oberhumer.com/opensource/lzo/download/ ? Don't know if ARJ opensource is truly the same as Jung's, but http://arj.sourceforge.net/files/arj-3.10.22.tar.gz is source to sourceforge page. ARC is another ancient tool, my first

  3. #3
    Member
    Join Date
    May 2008
    Location
    HK
    Posts
    160
    Thanks
    4
    Thanked 25 Times in 15 Posts
    You're talking lzh(lha) right?
    It does open sourced. http://git.sourceforge.jp/view?p=lha/lha.git
    And a rewritten design is located in http://git.sourceforge.jp/view?p=lha/olha.git
    yet another decompressor: https://github.com/fragglet/lhasa

  4. #4
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Thank you guys. By LZH I meant Lempel-Ziv-Huffman because I wanted a step up from super-fast compressors like LZO while maintaining very fast decompression.
    Bzip2 doesn't decompress nearly as fast as it's needed. LZO...I already have it, I just didn't mention it because in my head it's in another (faster) league.
    Arj - closed source.
    Arc - to check
    On a brief look, lha looks good.

  5. #5
    Member chornobyl's Avatar
    Join Date
    May 2008
    Location
    ua/kiev
    Posts
    153
    Thanks
    0
    Thanked 0 Times in 0 Posts

  6. #6
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    arjz is open-source: http://freearc.org/Research.aspx (arjz 0.50 is pretty close to rar 2.0 since i stealed all its tricks and added a few of my own)
    uc2 too
    cabarc-compatible archivers are available

  7. #7
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    @ m^2: what about the arj-link Karhunen provided above?
    @ Bulat: arjz015.zip-link is broken on http://freearc.org/Research.aspx

    Best regards!
    Last edited by Vacon; 5th March 2012 at 15:06. Reason: name corrected, sorry!

  8. #8
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by chornobyl View Post
    Already have this one. I considered this one to be fastest too.
    To prevent further confusion, I'll list all codecs that I or considered have already, including the faster ones:

    I have:
    brieflz (in progress)
    fastlz
    lz4
    lz4hc
    lzf
    lzjb
    lzmat
    lzo
    nrv2*
    quicklz
    snappy
    tornado
    zlib

    Know about (including the latest suggestions):
    -7-zip
    -arc
    -arj
    -arjz
    -cabarc
    -lha
    -lzham
    -lzrw*
    -NTFS-3G
    -uc2
    Quote Originally Posted by Bulat Ziganshin View Post
    arjz is open-source: http://freearc.org/Research.aspx (arjz 0.50 is pretty close to rar 2.0 since i stealed all its tricks and added a few of my own)
    uc2 too
    cabarc-compatible archivers are available
    Thanks. It seems I'm not as good searcher as I thought I am.
    And I didn't know there are open source cabarc implementations, this sure looks like a good option.
    Quote Originally Posted by Vacon View Post
    @ m^2: what about the arj-link Karhunen provided above?
    Oops, this was little over 7 am and I came here having 5 minutes before going to the work. Thank you for correction and thanks to Karhunen too!
    Last edited by m^2; 5th March 2012 at 23:33.

  9. #9
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    I took a look at all suggestions, and for quick additions, only arc looks OK:
    arjz, us2 - heavily single threaded, large effort to make usable
    lha, ulha - Unix only
    arj - x86 only
    As to Cabarc, I found 1 library, LZX_compress. It was OK, but by just looking at the interface design I see that it's poorly suitable for the task. I may add it anyway, but not now. I think I will search some more, maybe there are other libraries too.

  10. #10
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    lha=ar002. and i don't understand how compression algo can be unix-specific

    arjz/uc2 - put all global vars into class

  11. #11
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    lha=ar002. and i don't understand how compression algo can be unix-specific
    Algorithm can't but implementation - that's a different matter. Everything can be ported, but I just reckon that the cost would be high.
    And sometimes the results of porting would be irrelevant, like with snappy-c, which relies on Linux asm (unless I ripped the asm from Linux's guts too).
    Quote Originally Posted by Bulat Ziganshin View Post
    arjz/uc2 - put all global vars into class
    Yeah, I know. I may do it. It's not a small task though, because there are over 40 files with global variables scattered all around them and unknown number of references to them in each of these codecs. Again, it's just a matter of time, ~~50 hours / codec is much.
    ADDED:
    I still don't understand it. In the fastest category, I found 9 easy to use libraries and 1 codec that was a part of other sources, but trivially separable. Here I have really just 3 (zlib, tor and well, ucl belongs here too), each of them was complicated compared to an average fast codec and everything else is harder, usually much, much harder.
    Last edited by m^2; 6th March 2012 at 00:30.

  12. #12
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    just compare sizes. simplest lzss coder should be 10-20 lines long, while only huffman implementation in zlib is about 1000 lines. lzss is much simpler to implement - it's why a lot of non-professionals do it just for fun. for lzh, we see mainly "commercial", "professional" implementations, usually as part of archivers

    and i don't remember non-portable things in ar002 sources. at least i've used them in my dos/dos32/win32/os2 archiver - arjz was built from zlib match finder and ar002 match codec
    Last edited by Bulat Ziganshin; 6th March 2012 at 00:44.

  13. #13
    Member
    Join Date
    May 2008
    Location
    HK
    Posts
    160
    Thanks
    4
    Thanked 25 Times in 15 Posts
    you can just compile lha under mingw environment.

  14. #14
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    just compare sizes. simplest lzss coder should be 10-20 lines long, while only huffman implementation in zlib is about 1000 lines. lzss is much simpler to implement - it's why a lot of non-professionals do it just for fun. for lzh, we see mainly "commercial", "professional" implementations, usually as part of archivers
    True there are exceptions (LZO is by far the biggest codec that I have now), but LZSS is much simpler. Nevertheless there is no shortage of LZH / LZP(with entropy coding) / LZAri and similar codecs. It's just that they seem to be closed source more often. It may be that devs after having invested large effort hope for selling the rights.
    BTW, I wonder whether it's really a lot of work to write an open source LZH codec. You can use ready bits. There are good stand-alone match finders and entropy coders, one would have to write (or adapt) a parser and tune things up to work together.

    Quote Originally Posted by Bulat Ziganshin View Post
    and i don't remember non-portable things in ar002 sources. at least i've used them in my dos/dos32/win32/os2 archiver - arjz was built from zlib match finder and ar002 match codec
    lha.h imports some unix headers. I did not check how deep is the reliance, just noted that it existed. Like I said, up to now I took just a quick look.

    Quote Originally Posted by roytam1 View Post
    you can just compile lha under mingw environment.
    It's quite possible that I could, it does have the headers, but cross-compiler compatibility is important for me.
    MSVC, GCC and Clang are a must and I would really like to support Sun-CC at some point. I also successfully compiled it with llvm-gcc and would like to support other compilers too, though they have lower priority.
    There is a reason for this selection:
    -GCC, well, it doesn't need explanation
    -Clang. It's the default kernel compiler for FreeBSD and therefore I personally find it to be the most important.
    -Sun-CC. Solaris is built with it. AFAIK it can't be build with anything else yet.
    -MSVC. Barely interesting from filesystem perspective, but I care a lot about the benchmark being flexible, in particular it should work well as a general-purpose in-memory benchmark. And here MSVC is huge.

    I could add platform-specific codecs guarded by some #ifs, but I feel uneasy about it. I really want all platforms to be equally supported. I already thought a lot about it with regard to snappy-c and gave it a non-final rejection. Maybe at some point, but for now - no. So ATM it's either porting or skipping. So, TBD after a deeper look, but most likely, skipping.

  15. #15
    Member Karhunen's Avatar
    Join Date
    Dec 2011
    Location
    USA
    Posts
    91
    Thanks
    2
    Thanked 1 Time in 1 Post
    If you want some ancient stuff, remember the Lossless Compression Kit ? One of my first downloads on Lynx

  16. #16
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    In general, I'm not looking for compressors for historical reasons, but seeing how few algorithms in the speed range I'm exploring I managed to gather, I will sure look into it, thanks.

  17. #17
    Member
    Join Date
    Jan 2007
    Location
    Moscow
    Posts
    239
    Thanks
    0
    Thanked 3 Times in 1 Post
    Inspired by Snappy - Yappy. Russian page. http://blog.gamedeff.com/?p=371

  18. #18
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by nimdamsk View Post
    Inspired by Snappy - Yappy. Russian page. http://blog.gamedeff.com/?p=371
    Very interesting, thank you.

  19. #19
    Member Karhunen's Avatar
    Join Date
    Dec 2011
    Location
    USA
    Posts
    91
    Thanks
    2
    Thanked 1 Time in 1 Post
    Another one I remember about, used in a genealogy program, The SQX archiver, also supported by Total Commander and a program GUI that looks like Peazip, SimplyZip

  20. #20
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Thanks, though both seem to be closed source, therefore useless for me.
    In the meantime I did not take a deeper look into any already mentioned codec, aside from LZHAM (which is nicely integrated already, though w/out advanced options available from the command line).
    But I searched for more codecs. Some fast LZP algorithms can have decompression speed close to that of LZH, so I dug there. I added Charles Bloom's LZP1. Not great. I tried LZP2. Compression strength is very nice, but decompression crashes even in his sample program, compiled both with mingw 4.5.2 and mingw64 4.6.2.
    I tried flzp, but it's very bad. I guess that's because it was intended to be used as a preprocessor, not a standalone codec.
    I know that FreeArc has a LZP too. TOCHECK.
    I also decided to try alternative deflate implementations (minizip comes first) and to add LZX_compress, though it was not designed for memory to memory compression.
    ADDED: It just came to my mind that PeaZip that you mentioned uses a modified Deflate. I don't like the author, but well.. TOCHECK.
    ADDED: Checked PeaZip already.
    I hoped for a custom deflate-like algorithm, but found a wrapper around deflate that provides fast random access in a compressed stream. And written in Pascal. Well, not for me.
    Last edited by m^2; 13th March 2012 at 01:15.

  21. #21
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by m^2 View Post
    Thanks, though both seem to be closed source, therefore useless for me.[...]
    Not quite... It's sourcecode is open as you can read here:
    http://www.sqx-archiver.com/download.html
    This is where you can download source code, source samples and precompiled DLLs supplied by us and other developers.
    Afaik one is only allowed to use the source to develop SQX-Archivers / un-SQX-archivers, but not to develop derivats.
    But it's not developed anymore, because it's original author died some years ago (http://blog.speedproject.de/2009/08/...eufelskreis/):

    There has even been an attempt to port it to .Net on a command line
    Since I do not find it anymore on quickzip.org (maybe some other guy has more luck) I attach the binary, because it prints "released under a BSD-license".
    Edit: http://www.encode.su/threads/903-SQX-dead-born-star-!
    @ Bulat: sorry I wanted to compress it to arc-format, but forum is restricted

    Best regards!
    Attached Files Attached Files
    Last edited by Vacon; 13th March 2012 at 12:46. Reason: some additional remarks

  22. #22
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Vacon View Post
    Hello everyone,



    Not quite... It's sourcecode is open as you can read here:
    http://www.sqx-archiver.com/download.html

    Afaik one is only allowed to use the source to develop SQX-Archivers / un-SQX-archivers, but not to develop derivats.
    But it's not developed anymore, because it's original author died some years ago (http://blog.speedproject.de/2009/08/22/ein-teufelskreis/):

    There has even been an attempt to port it to .Net on a command line
    Since I do not find it anymore on quickzip.org (maybe some other guy has more luck) I attach the binary, because it prints "released under a BSD-license".
    Edit: http://www.encode.su/threads/903-SQX-dead-born-star-!
    @ Bulat: sorry I wanted to compress it to arc-format, but forum is restricted

    Best regards!
    Thanks for the suggestion. But I downloaded the source package. And the sources are only for:
    - decompression
    - of SQX1
    - on Windows
    There are some libs and dlls, maybe they do more, but they don't cut it for me....

  23. #23
    Member Vacon's Avatar
    Join Date
    May 2008
    Location
    Germany
    Posts
    523
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello everyone,

    Quote Originally Posted by m^2 View Post
    Thanks for the suggestion. But I downloaded the source package. And the sources are only for:
    - decompression
    - of SQX1
    - on Windows
    There are some libs and dlls, maybe they do more, but they don't cut it for me....
    One can like something or one can dislike something. Some things fit someone needs or some things don't. That's simple and that's life. Nothing to complain here.
    But this is an excerpt from ***\SQX-SDK_v206\Format\SQX1\sqx_note.txt:
    [...]The purpose of this document is to allow anyone to compress or decompress SQX
    compressed data. This document describes only the format of the output, it
    does not provide any specific algorithms for Huffmann coding, LZ77 coding
    etc., it is assumed that the reader is comfortable with the compression
    algorithms mentioned in this document.

    [...]

    It is the opposite for SQX: You are welcome to use the design specification
    and/or related source code/libs/dll's to create/edit/extract SQX archives freely
    as long as you respect my End User Licence: You must not claim that SQX is your
    property and you must not missuse my/our trade marks (see sqx-sdk.pdf).[...]
    And this is a cite from ***\SQX-SDK_v206\SQX-SDK.chm:
    [...]You may:

    - use the Software (including the redistributable code, sample code, source code and design specifications) free of charge in any of your applications provided you comply with this end-user license agreement;
    - install and use the Software on as many computers you use exclusively for the purpose of developing applications which create, extract or manipulate files that are compatible with SQX file format;
    - modify the sample source code to design, develop, and test your application. You may also reproduce and distribute the sample code in object code form along with any modifications you make to the sample code, provided that the modifications do not create files which are not compatible with the SQX file format;
    - modify the source code located in the Software's "Source" folder to design, develop, and test your application. You may also reproduce and distribute the source code in object code form along with any modifications you make to the source code, provided that the modifications do not create files which are not compatible with the SQX file format. [...]
    Note: the bolder line above has been made bold by me.
    Did I miss or misinterpreted something?

    Edit: on a sidenote: I found the SQX-Arc binary I attached yesterday inside the SQX-SDK 2.06

    Best regards!
    Last edited by Vacon; 14th March 2012 at 10:11. Reason: found .NET SQX-Arc in SQX-SDK :)

  24. #24
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,511
    Thanks
    746
    Thanked 668 Times in 361 Posts
    a few years ago i've studied SQX sources. afair there was nothing interesting in the published sources. thay are using LZH engine with 32 mb dictionary and MM preprocessing, but even such pretty limited engine was not published

    I wanted to compress it to arc-format, but forum is restricted
    for the good reson - imagine someone else trying to download the file. i don't like this forum restriction but only because freearc users sometimes need to send me their archives. otherwise it's better to compress to 7z at least (since it supported by winzip and winrar)
    Last edited by Bulat Ziganshin; 15th March 2012 at 21:26.

  25. #25
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Vacon View Post
    Hello everyone,


    One can like something or one can dislike something. Some things fit someone needs or some things don't. That's simple and that's life. Nothing to complain here.
    But this is an excerpt from ***\SQX-SDK_v206\Format\SQX1\sqx_note.txt:


    And this is a cite from ***\SQX-SDK_v206\SQX-SDK.chm:

    Note: the bolder line above has been made bold by me.
    Did I miss or misinterpreted something?

    Edit: on a sidenote: I found the SQX-Arc binary I attached yesterday inside the SQX-SDK 2.06

    Best regards!
    It's not that I don't like it, quite the opposite. At some point it was my primary compressor and I still have some sentiment to it. When I saw that it turned open source, I was really excited. That's why I was really disappointed to learn that it didn't.
    Yes, I read words like this too, but it doesn't change the fact that the sqx compression code is absent. Maybe by 'creating' they meant that the precompiled binaries allow it (I didn't check) or just changed their minds between writing these words and publishing the package, I don't know.
    Anyway, they code looks like a development snapshot of the decoding branch. And not the latest one, because some parts are missing (i.e. SQX2), some look like they weren't developed at the time of making the snapshot, for example there is a linux support file...but it's complete contents are:
    Code:
    //- :-(

  26. #26
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Found another fast, open source LZH:
    http://sites.google.com/site/compgt/lz77
    But it's meant for educational purposes, my initial evaluation shows that it's simple but inefficient and therefore not really interesting for me...

Similar Threads

  1. Open source JPEG compressors
    By inikep in forum Data Compression
    Replies: 8
    Last Post: 22nd October 2011, 01:16
  2. Non open source Data compression Tools
    By ehsansad in forum Data Compression
    Replies: 9
    Last Post: 22nd September 2011, 01:41
  3. PeaZip - open source archiver
    By squxe in forum Data Compression
    Replies: 1
    Last Post: 3rd December 2009, 22:01
  4. fcm1 - open source order-1 cm encoder
    By encode in forum Data Compression
    Replies: 34
    Last Post: 5th June 2008, 00:16
  5. New fast open-source paq-based jpeg compressor
    By Bulat Ziganshin in forum Forum Archive
    Replies: 14
    Last Post: 13th September 2007, 14:57

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •