Results 1 to 14 of 14

Thread: zling updates

  1. #1
    Member RichSelian's Avatar
    Join Date
    Aug 2011
    Location
    Shenzhen, China
    Posts
    160
    Thanks
    18
    Thanked 56 Times in 27 Posts

    Talking zling updates

    tag: https://github.com/richox/zling/archive/cristmas.zip
    homepage: https://github.com/richox/zling

    rewrote in C++ for taking a codemaster authorization in my company...
    still lightweight (about 1k lines of source code) with good performance. now it has much better ratio and encode/decode speed than gzip.

    waiting for benchmarks and bug reports

  2. The Following 6 Users Say Thank You to RichSelian For This Useful Post:

    cade (3rd February 2014),lorents17 (4th October 2015),Matt Mahoney (27th December 2013),Nania Francesco (27th December 2013),Paul W. (30th December 2013),samsat1024 (26th December 2013)

  3. #2
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Please, if possible, put executable for download !

  4. #3
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    http://mattmahoney.net/dc/text.html#2824
    Nice improvement and now makes the Pareto frontier for compression time.

    I had to install Boost to compile so I only tested in Linux. I suggest using stdint.h instead of boost/cstdint.hpp because not everyone has Boost already installed.

  5. #4
    Member
    Join Date
    Nov 2011
    Location
    Tunisia
    Posts
    8
    Thanks
    155
    Thanked 4 Times in 4 Posts
    Generic 32 and 64 windows binaries compiled with MSYS2(with pacman)/gcc 4.8.2 threads-posix, (64-bit:seh) / (32-bit:dwarf2)

    Checksums :

    File;crc32;md5;sha1;sha2 (256)
    Code:
    zling_bin-git-27-12-2013.zip;48F77ED4;4AF8DE232F6CC99066DC2BCC10A3FCC2;3AEB6D241D87087026B4674172F891084B81DC00;B5D4BB9689C1C05BA787A0F15531DCB846467A30AC9F6D499851C57739D9E863
    zling32.exe;50391926;B7B22BDB2ACA0FB8D1891EB979EEAE66;0371AB04EC2EE4D83B01FB470ADF1FB3FBFD35DD;7B13E4B8C19CB996761CC7123A93D32AB97EEC8AF98CBD3AA9A5899D6626CA8D
    zling64.exe;CF446944;6F4B901E318F36FB71C8E2CB2A8AC811;105425B3F7BBFB09672EC5E3BB006351E2F0BA60;EF2C79C383780AC239B54AB7FA854A374949999853AEE25132B8BEECF934929B
    Attached Files Attached Files

  6. The Following User Says Thank You to samsat1024 For This Useful Post:

    Nania Francesco (28th December 2013)

  7. #5
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    776
    Thanks
    63
    Thanked 270 Times in 190 Posts
    Input RAM-disk:
    1,875,119,714 bytes, DB index

    Output RAM-disk:
    856,832,655 bytes, 95.139 sec. - 12.305 sec., gzip
    871,083,745 bytes, 43.238 sec. - 14.176 sec., zlite64
    818,579,962 bytes, 44.372 sec. - 12.906 sec., zling64

    ---------------

    Input RAM-disk:
    927,609,577 bytes, log file mainly text

    Output RAM-disk:
    170,186,586 bytes, 19.805 sec. - 4.131 sec., gzip
    105,545,481 bytes, 7.416 sec. - 2.854 sec., zlite64
    89,426,749 bytes, 5.391 sec. - 1.823 sec., zling64

    ---------------

    Input RAM-disk:
    916,873,928 bytes, html file

    Output RAM-disk:
    168,435,607 bytes, 19.351 sec. - 4.075 sec., gzip
    163,979,454 bytes, 10.021 sec. - 3.707 sec., zlite64
    153,168,791 bytes, 8.634 sec. - 2.563 sec., zling64

    ---------------

    Input RAM-disk:
    937,345,926 bytes, IIS log file

    Output RAM-disk:
    60,100,354 bytes, 9.402 sec. - 2.865 sec., gzip
    34,752,715 bytes, 3.653 sec. - 1.586 sec., zlite64
    33,648,395 bytes, 2.512 sec. - 0.971 sec., zling64

  8. #6
    Member
    Join Date
    Jun 2013
    Location
    USA
    Posts
    98
    Thanks
    4
    Thanked 14 Times in 12 Posts
    Doesn't perform better than gzip -9 on rockyou.txt filesize wise. It does compress faster though.

    53350182 vs.

    58602467

    former is gzip.

  9. #7
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    WCC test not passed in TAR mode (APP4) and single file mode (APP1, program go in crush !)
    Last edited by Nania Francesco; 28th December 2013 at 12:44.

  10. The Following User Says Thank You to Nania Francesco For This Useful Post:

    RichSelian (28th December 2013)

  11. #8
    Member RichSelian's Avatar
    Join Date
    Aug 2011
    Location
    Shenzhen, China
    Posts
    160
    Thanks
    18
    Thanked 56 Times in 27 Posts
    Quote Originally Posted by Nania Francesco View Post
    WCC test not passed in TAR mode (APP4) and single file mode (APP1, program go in crush !)
    thanks very much for the bug report. the bug happens in an extreme case when there's only one match found in a block.
    already fixed: https://github.com/richox/zling/arch...as_bugfix1.zip

  12. The Following User Says Thank You to RichSelian For This Useful Post:

    samsat1024 (29th December 2013)

  13. #9
    Member
    Join Date
    Nov 2011
    Location
    Tunisia
    Posts
    8
    Thanks
    155
    Thanked 4 Times in 4 Posts
    Generic 32 and 64 windows binaries compiled with MSYS2(with pacman)/gcc 4.8.2 threads-posix, (64-bit:seh) / (32-bit:dwarf2)

    Checksums :

    File;crc32;md5;sha1;sha2 (256)
    Code:
    zling_bin-git-28-12-2013.zip;AD20E2A6;4B6AF22B66390B3425C3FFB73A791692;48DB3B32E965ADA0BC54C81870A63659A09BDF5A;3105EEB38D339BA1582084D8F87347865FD5A5711AFBAEB0EB52D55802AB1BD1
    zling32.exe;1D028B3D;0C0C9C06E8AFA076641A059A17848EE5;137D850AD50466DCA47A7BB5784B598FA67E251E;0114E939556FC1762B2752CB42718ED7CB581CD3CF5E677A4743B5F319724B1D
    zling64.exe;DB44B35F;1D5E6D9FAC19569769FC1BC8660D631F;CE3FC5D70B2B54CFD11C38CD69B2FC8BC9D574D3;A0EA99C2DE3E7100716EE18BB470E877D6D4721BE613D7607C018FD59222E9C0
    Attached Files Attached Files
    Last edited by samsat1024; 29th December 2013 at 01:29.

  14. The Following User Says Thank You to samsat1024 For This Useful Post:

    load (29th December 2013)

  15. #10
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    WCC benchmark OK !

  16. #11
    Member RichSelian's Avatar
    Join Date
    Aug 2011
    Location
    Shenzhen, China
    Posts
    160
    Thanks
    18
    Thanked 56 Times in 27 Posts
    updated to: https://github.com/richox/zling/tree/20140121
    even faster compression and boost is wiped out!

    Quote Originally Posted by Nania Francesco View Post
    Please, if possible, put executable for download !
    Sorry but I'm working on a server of my company, which has only an old gcc-3.4.5 compiler. I don't have other compilers installed on my laptop (I don't like to write codes on suck Windows 7
    The source code is easy to compile with gcc/mingw or clang. no external libs are needed. even `gcc -Wall -O3 *.cpp` is OK.

  17. The Following User Says Thank You to RichSelian For This Useful Post:

    samsat1024 (21st January 2014)

  18. #12
    Member ivan2k2's Avatar
    Join Date
    Nov 2012
    Location
    Russia
    Posts
    35
    Thanks
    13
    Thanked 6 Times in 3 Posts
    Compiled zling for Windows with mingw 4.8.1 with -O3 -s -static options, 32 and 64 bit.
    Attached Files Attached Files

  19. The Following 4 Users Say Thank You to ivan2k2 For This Useful Post:

    avitar (24th January 2014),Matt Mahoney (21st January 2014),samsat1024 (21st January 2014),surfersat (22nd January 2014)

  20. #13
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts

  21. #14
    Member
    Join Date
    Jul 2013
    Location
    Stanford, California
    Posts
    24
    Thanks
    7
    Thanked 2 Times in 2 Posts
    Hi,

    I'm wondering what effort would be needed to use zling or zlite to just test for a tight upper bound on compressed size. Because of the fast compression speed, these algorithms could get at that information pretty quickly. And because of the noteworthy compression ratio attainable at that speed, the bound or estimate arrived at can be a more realistic bound. I understand there are other ways to test whether data is compressible at all, and also that there are other algorithms to choose from given tradeoffs in ratio and speed. But I'd like to estimate how compressible certain data is, and I'd like to do that fairly quickly, so I'm wondering about zling/zlite in particular. The modification of the algorithm to just do the estimate might suggest some simplifications. For example, it potentially wouldn't be necessary to allocate a subset of the memory that is currently being allocated or used by zling/zlite (since the data output would not be used), and likewise maybe some bytes wouldn't need to be copied or processed if the aim is just the estimate. I'd like to know what would be necessary to prune the algorithm down to forming an estimate on compressed size. Initially the estimate could be exact (a nearly identical iteration of the algorithm), but a probabilistic compromise would suffice too (for example if the bound is within 5% of the actual value with >= 95% probability). As a next iteration, I'd like to be able to incrementally process an input buffer of, for example, 32MB, and every 2MB I'd like to reassess the estimate on compressed size, even though this estimate need not flush the output at those boundaries. This is to better assess correlation of the data.

    If I ask for the estimate on the first 4MB and that gives 2.5MB, but the estimate for the first 8MB total gives 3.5MB (and an estimate on the 4MB to 8MB range alone gives 2.5MB), then there's noted correlation or commonality between the first 4MB and the next. Similarly, if the estimate for the first 8MB total gave 6.5MB, then not only would the second 4MB be likely to be uncorrelated, but it would be estimated to be incompressible.

    Any thoughts or ideas would be appreciated.
    Last edited by Intensity; 24th January 2014 at 17:45.

Similar Threads

  1. zpaq updates
    By Matt Mahoney in forum Data Compression
    Replies: 2527
    Last Post: 4th May 2019, 12:33
  2. GUI winzpaq updates
    By Sportman in forum Data Compression
    Replies: 31
    Last Post: 3rd August 2013, 23:21
  3. comprox/comprolz updates
    By RichSelian in forum Data Compression
    Replies: 21
    Last Post: 18th December 2012, 21:31
  4. comprox updates
    By RichSelian in forum Data Compression
    Replies: 1
    Last Post: 5th November 2011, 23:19
  5. Metacompressor.com benchmark updates
    By Sportman in forum Data Compression
    Replies: 79
    Last Post: 22nd April 2009, 03:24

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •