zling updates

[RichSelian]

tag: https://github.com/richox/zling/archive/cristmas.zip
homepage: https://github.com/richox/zling

rewrote in C++ for taking a codemaster authorization in my company...
still lightweight (about 1k lines of source code) with good performance. now it has much better ratio and encode/decode speed than gzip.

waiting for benchmarks and bug reports

[Nania Francesco]

Please, if possible, put executable for download !

[Matt Mahoney]

http://mattmahoney.net/dc/text.html#2824
Nice improvement and now makes the Pareto frontier for compression time.

I had to install Boost to compile so I only tested in Linux. I suggest using stdint.h instead of boost/cstdint.hpp because not everyone has Boost already installed.

[samsat1024]

Generic 32 and 64 windows binaries compiled with MSYS2(with pacman)/gcc 4.8.2 threads-posix, (64-bit:seh) / (32-bit:dwarf2)

Checksums :

File;crc32;md5;sha1;sha2 (256)

zling_bin-git-27-12-2013.zip;48F77ED4;4AF8DE232F6CC99066DC2BCC10A3FCC2;3AEB6D241D87087026B4674172F891084B81DC00;B5D4BB9689C1C05BA787A0F15531DCB846467A30AC9F6D499851C57739D9E863
zling32.exe;50391926;B7B22BDB2ACA0FB8D1891EB979EEAE66;0371AB04EC2EE4D83B01FB470ADF1FB3FBFD35DD;7B13E4B8C19CB996761CC7123A93D32AB97EEC8AF98CBD3AA9A5899D6626CA8D
zling64.exe;CF446944;6F4B901E318F36FB71C8E2CB2A8AC811;105425B3F7BBFB09672EC5E3BB006351E2F0BA60;EF2C79C383780AC239B54AB7FA854A374949999853AEE25132B8BEECF934929B

[Sportman]

Input RAM-disk:
1,875,119,714 bytes, DB index

Output RAM-disk:
856,832,655 bytes, 95.139 sec. - 12.305 sec., gzip
871,083,745 bytes, 43.238 sec. - 14.176 sec., zlite64
818,579,962 bytes, 44.372 sec. - 12.906 sec., zling64

---------------

Input RAM-disk:
927,609,577 bytes, log file mainly text

Output RAM-disk:
170,186,586 bytes, 19.805 sec. - 4.131 sec., gzip
105,545,481 bytes, 7.416 sec. - 2.854 sec., zlite64
89,426,749 bytes, 5.391 sec. - 1.823 sec., zling64

---------------

Input RAM-disk:
916,873,928 bytes, html file

Output RAM-disk:
168,435,607 bytes, 19.351 sec. - 4.075 sec., gzip
163,979,454 bytes, 10.021 sec. - 3.707 sec., zlite64
153,168,791 bytes, 8.634 sec. - 2.563 sec., zling64

---------------

Input RAM-disk:
937,345,926 bytes, IIS log file

Output RAM-disk:
60,100,354 bytes, 9.402 sec. - 2.865 sec., gzip
34,752,715 bytes, 3.653 sec. - 1.586 sec., zlite64
33,648,395 bytes, 2.512 sec. - 0.971 sec., zling64

[Mangix]

Doesn't perform better than gzip -9 on rockyou.txt filesize wise. It does compress faster though.

53350182 vs.

58602467

former is gzip.

[Nania Francesco]

WCC test not passed in TAR mode (APP4) and single file mode (APP1, program go in crush !)

[RichSelian]

WCC test not passed in TAR mode (APP4) and single file mode (APP1, program go in crush !)

thanks very much for the bug report. the bug happens in an extreme case when there's only one match found in a block.
already fixed: https://github.com/richox/zling/arch...as_bugfix1.zip

[samsat1024]

Generic 32 and 64 windows binaries compiled with MSYS2(with pacman)/gcc 4.8.2 threads-posix, (64-bit:seh) / (32-bit:dwarf2)

Checksums :

File;crc32;md5;sha1;sha2 (256)

zling_bin-git-28-12-2013.zip;AD20E2A6;4B6AF22B66390B3425C3FFB73A791692;48DB3B32E965ADA0BC54C81870A63659A09BDF5A;3105EEB38D339BA1582084D8F87347865FD5A5711AFBAEB0EB52D55802AB1BD1
zling32.exe;1D028B3D;0C0C9C06E8AFA076641A059A17848EE5;137D850AD50466DCA47A7BB5784B598FA67E251E;0114E939556FC1762B2752CB42718ED7CB581CD3CF5E677A4743B5F319724B1D
zling64.exe;DB44B35F;1D5E6D9FAC19569769FC1BC8660D631F;CE3FC5D70B2B54CFD11C38CD69B2FC8BC9D574D3;A0EA99C2DE3E7100716EE18BB470E877D6D4721BE613D7607C018FD59222E9C0

[Nania Francesco]

WCC benchmark OK !

[RichSelian]

updated to: https://github.com/richox/zling/tree/20140121
even faster compression and boost is wiped out!

Please, if possible, put executable for download !

Sorry but I'm working on a server of my company, which has only an old gcc-3.4.5 compiler. I don't have other compilers installed on my laptop (I don't like to write codes on suck Windows 7
The source code is easy to compile with gcc/mingw or clang. no external libs are needed. even `gcc -Wall -O3 *.cpp` is OK.

[ivan2k2]

Compiled zling for Windows with mingw 4.8.1 with -O3 -s -static options, 32 and 64 bit.

[Matt Mahoney]

LTCB: http://mattmahoney.net/dc/text.html#2818
Silesia: http://mattmahoney.net/dc/silesia.html

[Intensity]

Hi,

I'm wondering what effort would be needed to use zling or zlite to just test for a tight upper bound on compressed size. Because of the fast compression speed, these algorithms could get at that information pretty quickly. And because of the noteworthy compression ratio attainable at that speed, the bound or estimate arrived at can be a more realistic bound. I understand there are other ways to test whether data is compressible at all, and also that there are other algorithms to choose from given tradeoffs in ratio and speed. But I'd like to estimate how compressible certain data is, and I'd like to do that fairly quickly, so I'm wondering about zling/zlite in particular. The modification of the algorithm to just do the estimate might suggest some simplifications. For example, it potentially wouldn't be necessary to allocate a subset of the memory that is currently being allocated or used by zling/zlite (since the data output would not be used), and likewise maybe some bytes wouldn't need to be copied or processed if the aim is just the estimate. I'd like to know what would be necessary to prune the algorithm down to forming an estimate on compressed size. Initially the estimate could be exact (a nearly identical iteration of the algorithm), but a probabilistic compromise would suffice too (for example if the bound is within 5% of the actual value with >= 95% probability). As a next iteration, I'd like to be able to incrementally process an input buffer of, for example, 32MB, and every 2MB I'd like to reassess the estimate on compressed size, even though this estimate need not flush the output at those boundaries. This is to better assess correlation of the data.

If I ask for the estimate on the first 4MB and that gives 2.5MB, but the estimate for the first 8MB total gives 3.5MB (and an estimate on the 4MB to 8MB range alone gives 2.5MB), then there's noted correlation or commonality between the first 4MB and the next. Similarly, if the estimate for the first 8MB total gave 6.5MB, then not only would the second 4MB be likely to be uncorrelated, but it would be estimated to be incompressible.

Any thoughts or ideas would be appreciated.