_FOSSIL_: interesting test file

[m^2]

I have a test file that's often shows differences between codecs in the same class. It's a fossil SCM repository database.
Example results, gcc 4.7.2, Core 2Duo@2.66 ghz:

density::chameleon                      2015-01-29   
    1912308 (x 1.815)      336 MB/s  967 MB/s       150e6  434e6
density::mandala                        2015-01-29   
     421772 (x 8.230)      803 MB/s  619 MB/s       705e6  543e6
LZ4                                     r127         
     211289 (x16.429)     1062 MB/s 1326 MB/s       997e6 1244e6
LZO                                     2.06         1x1
     218435 (x15.892)     1108 MB/s  587 MB/s      1038e6  549e6
QuickLZ                                 1.5.1b6      1
     212944 (x16.302)      583 MB/s  914 MB/s       547e6  857e6
Snappy                                  1.1.0        
     338751 (x10.248)      996 MB/s 1165 MB/s       898e6 1051e6
ZSTD                                    0.0.1        
    3471360 (x 1.000)     1174 MB/s         -         0e0    0e0
lrrle                                   0            256
     602832 (x 5.758)     6826 MB/s 6105 MB/s      5640e6 5039e6
blosc                                   1.2.3        5
     498123 (x 6.969)      775 MB/s 3191 MB/s       663e6 2733e6
LZF                                     3.6          
     233019 (x14.897)      229 MB/s  440 MB/s       213e6  410e6
LZFX                                    r16          
     236427 (x14.683)      588 MB/s  424 MB/s       548e6  394e6
Yappy                                   v2           10
     616793 (x 5.628)     37.1 MB/s 1220 MB/s        30e6 1003e6

Here we go:
density::chameleon is extremely week.
density::mandala isn't obviously wrong, but seems to have lost its superb efficiency.
LZ4 and LZO work great.
QuickLZ is rather slow.
Snappy has good speed, but is nowhere near as strong as LZ4 / LZO.
ZSTD fails to encode it.
lrrle works great, it's rare.
blosc works great, it's rate too.
LZFX is overall better than LZF, I haven't seen it elsewhere.
Yappy is very weak.





ADDED: I screwed up, upladed a wrong file, the right one will arrive after the weekend. Sorry.

[Kennon Conrad]

The unzipped file is only 442,368 bytes. Shouldn't it be 3,471,360 bytes?

[m^2]

I screwed up, upladed a wrong file, the right one will arrive after the weekend. Sorry.

[m^2]

Uploaded the right file.

[Kennon Conrad]

I tried Tree64 v0.19 (to be released soon) and got 107,612 bytes (x 32.258). Compression is very slow but decompression is faster than I can accurately measure with Timer64. It's maybe 400 MB/sec on decompression.

[m^2]

Have you thought about turning Tree to a library?

[Kennon Conrad]

Have you thought about turning Tree to a library?

Not until you mentioned it. It seems like a good idea. I looked at Fsbench and see you are the author. How are the libraries linked? Is it static linking at compile time or something more complex? If that's all it is, it seems like it should be a fairly simple task.

[xezz]

I tested xeloz and repair.

xeloz c6 -> 156563
xeloz c68 -> 151722
repair -> 136724

[m^2]

Not until you mentioned it. It seems like a good idea. I looked at Fsbench and see you are the author. How are the libraries linked? Is it static linking at compile time or something more complex? If that's all it is, it seems like it should be a fairly simple task.

Simple static linking at the source file level, that is - I don't link stuff into libraries and then into a final executable but instead I compile everything a do a link of everything.
Though I may change it, now I have problems when 2 libraries have symbol with the same names, it has just happened for the second time.