Results 1 to 5 of 5

Thread: My examples like lz4 & lz5

  1. #1
    Member lz77's Avatar
    Join Date
    Jan 2016
    Location
    Russia
    Posts
    46
    Thanks
    14
    Thanked 11 Times in 7 Posts

    My examples like lz4 & lz5

    Sorry, I'm not a profi programmer and not an English speaker.
    I've read about lz4 & lz5 by Yann Collet and downloaded his examples from Github (lz4.exe & lz5.exe). I added several improvements in lz77 algorithm and wrote Win32 example, which shows compression ratio on ~ 4% better than lz5 with simple hash.

    My program uses simple hash and hash table on 1M cells like lz5.exe (as I saw it). I've compared my program with lz5 in simple hash mode on a 800Kb plain text file and a 8.3Mb exe file.

    I want to know is anyone interested in this info?

    Remark: my example stores compressed data in raw format.
    Last edited by lz77; 19th January 2016 at 16:36.

  2. Thanks (3):

    Bulat Ziganshin (19th January 2016),Cyan (19th January 2016),Stephan Busch (19th January 2016)

  3. #2
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    865
    Thanks
    463
    Thanked 260 Times in 107 Posts
    Continue your interesting investigations, and at some point, you will find your results in good enough shape to be shared.

    Always check your compressed stream can be decoded with appropriate source code.
    There is always a risk that compressed format proved not decodable if done later on.

    Also : lz5 is work from @inikep.

  4. #3
    Member
    Join Date
    Nov 2015
    Location
    boot ROM
    Posts
    92
    Thanks
    27
    Thanked 15 Times in 14 Posts
    ...and on side note, -dev branch of LZ5 comes with a little surprise: it comes with a bit better ratio, especially on highest levels, so it also happens to be moving target . One can get even better ratio on large chunks of data by increasing sliding window size (but it forces one to keep larger buffer in memory and does not works for small chunks of data). Or by tweaking levels for better match finding (but it quickly gets slow, while gain is quite small).

    And IIRC, LZ5 is all about 1 << 22 window, which is 4Mb, not 1Mb (technical limit is 1 << 24). It also uses somewhat more complicated (vs LZ4) stream encoding, using "huffman like" codes (without doing any huffman of course). It gives more flexibility while trying not to lose speed too much. Overall LZ5 is somewhat slower than LZ4, but still comparable with most of "fast LZ" kind of things in general. While it can reach zlib-like ratios. In some selected cases it can actually beat zlib due to larger window, which looks funny to my taste, granted decompression speed is like 2-3x faster.
    Last edited by xcrh; 19th January 2016 at 19:51. Reason: forum dislikes "shift left" operator and screwed message up :(

  5. #4
    Member lz77's Avatar
    Join Date
    Jan 2016
    Location
    Russia
    Posts
    46
    Thanks
    14
    Thanked 11 Times in 7 Posts
    Thanks, I have a few more questions:
    1. Can I make money on such things? Or its just for fame/fun? I "live" in a bad region and have income only from Internet. So I'm looking for remote work.
    2. What development environment for C (with integrated debugger) is most suitable for the development of such programs?

    By the way: Jan 19 I received Chinese fanless PC. I think, in this store prices are cheapest:
    aliexpress.com/item/Fanless-mini-pc-5thCPU-Intel-Corei3-5005U-Broadwell-Barebone-NUC-PC-Fanless-HDMI-Intel-HM97-HD5500/32398772754.html

  6. #5
    Member lz77's Avatar
    Join Date
    Jan 2016
    Location
    Russia
    Posts
    46
    Thanks
    14
    Thanked 11 Times in 7 Posts
    My small test lile LZ4 is available for download, see http://encode.su/threads/2452-My-sma...by-Yann-Collet
    My test like lz5 will be soon.

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •