Is there any cruncher with statistical output?

[Crush]

Is there some kind of "experimental" compressor that gives statistical information on how often some special byte combinations repeat or how often something cannot be compressed and how long those blocks are, perhaps also what different methods can be applied or not like markov models, wave-offsets or other compressable bit-patterns? Google did not help to find anything usable. Do all compressor coders make their own test suites alone again or only code down their ideas and look how it´s getting? Any idea? The only thing that offers some data analysis I found is Cryptool 2.

[Kennon Conrad]

Have you looked at GLZA/Tree on the download page? It is an experimental compressor that gives statistical information on how many times the byte combinations it uses appear (by using the -v command line option when running GLZAencode), including literals. The first appearance of byte combinations are marked with an escape symbol and a length code and then put in a dictionary and assigned a dictionary code. It uses Markov modeling to determine which combinations are profitable and should be put in the dictionary. The latest version produces a compressed enwik9 file that is a little more than 14% smaller than the leading LZ based compression algorithm shown on the Large Text Compression Benchmark.

[Crush]

Thank you for the hint. GLZAencode was hidden in the Tree thread. This is a bit too simple only counting the amount of literals. Cryptool gives much better output with 2byte and 3byte amounts, but without any compression view. It would be more interesting to show standard RLE compressor behaviour showing how often also more complex strings appear and how often uncompressed blocks have to be signed.

[Kennon Conrad]

GLZAencode counts the symbols that are in the input file. It is part of a toolset (with example .bat files included) and the intent is that a compressed file is created by running GLZAformat, GLZAcompress and then GLZAencode. If either of the first two steps are skipped, the output will only contain literals (with one missing literal and possible other problems if the GLZAformat step is skipped). The statistical output for GLZAencode when compressing enwik9 with the intended sequence starts like this, with 'C' indicating the next letter should be capitalized:

0: #400960 L8: " "
1: #326425 L8: ","
2: #304387 L8: " C"
3: #295356 L8: "]]"
4: #219757 L8: " and"
5: #197922 L9: "s"
6: #174778 L9: ")"
7: #157298 L9: "|C"
8: #142221 L9: "]]
*[[C"
9: #141000 L9: ", C"
10: #137467 L9: " [[C"
11: #118636 L9: "]], [[C"
12: #108478 L9: "]],"
13: #107463 L9: ", and"
14: #101665 L9: "]]</text>
</revision>
</page>
<page>
<title>C"
15: #99892 L10: "&quot;"
16: #92969 L10: ". C"
17: #92332 L10: "]]
* [[C"
18: #86298 L10: " [["
19: #84405 L10: "''"
20: #74947 L10: " and C"
21: #71963 L10: "|"
22: #69614 L10: "]] and [[C"
23: #67163 L10: " or"
24: #55365 L10: " of C"
25: #54381 L10: "'s"
26: #53897 L10: ". Cthe"
27: #53476 L10: " is"
28: #52816 L10: " in"
29: #51935 L10: " of"
30: #51130 L10: ", the"

Symbols are shown from most common to least common with # showing how often the symbol for the corresponding string (dictionary entry) is used, the L is the length of the dictionary code in bits. The symbols are a mix of literals (1 byte except extended UTF-8 characters) and deduplicated longer strings created by GLZAcompress. The length of the strings represented by the dictionary symbols can be quite long, with the longest created for enwik9 representing a string that is 25,650 bytes long uncompressed (25,650 space characters, 0x20):

7412600: #2 L24: "25,640 spaces (0x20)"

GLZA is a complex (some would say convoluted) grammar compressor. It doesn't break things into "compressible" and "uncompressible" blocks, but you have given me something to think about. It just tries to find the lowest order 0 entropy grammar it can (GLZAcompress) and writes the final output using that grammar (GLZAencode). For text files, it is typical that almost everything is compressible, so I'm not sure how much breaking things into compressible/uncompressible blocks would help. I have found DNA to be trickier. GLZAcompress does pretty well at finding long matches (actually outperformed DNASequitor in a limited test) but the encoding could still use some improvement and the idea of identifying non-deduplicatible sections could be useful.

[Crush]

GLZA is some kind of dictionary compression. This is not the type of statistics I´m looking for, but better than nothing. I´m more aiming at some kind of LZ77 compression stats.
I made something similar but much much simpler than GLZA for a coder contest here:
An even more improved version is here: http://www.donationcoder.com/forum/i...?topic=21290.0
A fixed version from the contest with extended dictionaries is here (the .dic files are the compressed libraries - sourcecode is included): http://netpan.ironbytes.de/stuff/Salatschleuder.zip

[Kennon Conrad]

GLZA is some kind of dictionary compression. This is not the type of statistics I´m looking for, but better than nothing. I´m more aiming at some kind of LZ77 compression stats.

I'm a bit confused. As far as I know LZ77 compression stats wouldn't provide any statistical information on how often some special byte combinations repeat.

[Crush]

It´s about how often matches appear, how long they are and how much of the data cannot be compressed. The strings themselves are not so important for me.

[Lucas]

couldn't you just take an open source (LZ) compressor and add your own variables for counting how often literals are written out or matched? I don't imagine that'd be too difficult.

[Crush]

Yes, I think that´s the only solution.