Results 1 to 9 of 9

Thread: _FOSSIL_: interesting test file

  1. #1
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,610
    Thanks
    30
    Thanked 65 Times in 47 Posts

    _FOSSIL_: interesting test file

    I have a test file that's often shows differences between codecs in the same class. It's a fossil SCM repository database.
    Example results, gcc 4.7.2, Core 2Duo@2.66 ghz:

    Code:
    density::chameleon                      2015-01-29   
        1912308 (x 1.815)      336 MB/s  967 MB/s       150e6  434e6
    density::mandala                        2015-01-29   
         421772 (x 8.230)      803 MB/s  619 MB/s       705e6  543e6
    LZ4                                     r127         
         211289 (x16.429)     1062 MB/s 1326 MB/s       997e6 1244e6
    LZO                                     2.06         1x1
         218435 (x15.892)     1108 MB/s  587 MB/s      1038e6  549e6
    QuickLZ                                 1.5.1b6      1
         212944 (x16.302)      583 MB/s  914 MB/s       547e6  857e6
    Snappy                                  1.1.0        
         338751 (x10.248)      996 MB/s 1165 MB/s       898e6 1051e6
    ZSTD                                    0.0.1        
        3471360 (x 1.000)     1174 MB/s         -         0e0    0e0
    lrrle                                   0            256
         602832 (x 5.758)     6826 MB/s 6105 MB/s      5640e6 5039e6
    blosc                                   1.2.3        5
         498123 (x 6.969)      775 MB/s 3191 MB/s       663e6 2733e6
    LZF                                     3.6          
         233019 (x14.897)      229 MB/s  440 MB/s       213e6  410e6
    LZFX                                    r16          
         236427 (x14.683)      588 MB/s  424 MB/s       548e6  394e6
    Yappy                                   v2           10
         616793 (x 5.628)     37.1 MB/s 1220 MB/s        30e6 1003e6
    Here we go:
    density::chameleon is extremely week.
    density::mandala isn't obviously wrong, but seems to have lost its superb efficiency.
    LZ4 and LZO work great.
    QuickLZ is rather slow.
    Snappy has good speed, but is nowhere near as strong as LZ4 / LZO.
    ZSTD fails to encode it.
    lrrle works great, it's rare.
    blosc works great, it's rate too.
    LZFX is overall better than LZF, I haven't seen it elsewhere.
    Yappy is very weak.





    ADDED: I screwed up, upladed a wrong file, the right one will arrive after the weekend. Sorry.
    Attached Files Attached Files
    Last edited by m^2; 2nd February 2015 at 18:41.

  2. Thanks (2):

    Cyan (30th January 2015),Kennon Conrad (2nd February 2015)

  3. #2
    Member
    Join Date
    Jan 2014
    Location
    Bothell, Washington, USA
    Posts
    695
    Thanks
    153
    Thanked 183 Times in 108 Posts
    The unzipped file is only 442,368 bytes. Shouldn't it be 3,471,360 bytes?

  4. #3
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,610
    Thanks
    30
    Thanked 65 Times in 47 Posts
    I screwed up, upladed a wrong file, the right one will arrive after the weekend. Sorry.

  5. #4
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,610
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Uploaded the right file.

  6. #5
    Member
    Join Date
    Jan 2014
    Location
    Bothell, Washington, USA
    Posts
    695
    Thanks
    153
    Thanked 183 Times in 108 Posts
    I tried Tree64 v0.19 (to be released soon) and got 107,612 bytes (x 32.258). Compression is very slow but decompression is faster than I can accurately measure with Timer64. It's maybe 400 MB/sec on decompression.

  7. #6
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,610
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Have you thought about turning Tree to a library?

  8. #7
    Member
    Join Date
    Jan 2014
    Location
    Bothell, Washington, USA
    Posts
    695
    Thanks
    153
    Thanked 183 Times in 108 Posts
    Quote Originally Posted by m^2 View Post
    Have you thought about turning Tree to a library?
    Not until you mentioned it. It seems like a good idea. I looked at Fsbench and see you are the author. How are the libraries linked? Is it static linking at compile time or something more complex? If that's all it is, it seems like it should be a fairly simple task.

  9. #8
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    164
    Thanks
    31
    Thanked 64 Times in 40 Posts
    I tested xeloz and repair.
    Code:
    xeloz c6 -> 156563
    xeloz c68 -> 151722
    repair -> 136724

  10. #9
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,610
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Kennon Conrad View Post
    Not until you mentioned it. It seems like a good idea. I looked at Fsbench and see you are the author. How are the libraries linked? Is it static linking at compile time or something more complex? If that's all it is, it seems like it should be a fairly simple task.
    Simple static linking at the source file level, that is - I don't link stuff into libraries and then into a final executable but instead I compile everything a do a link of everything.
    Though I may change it, now I have problems when 2 libraries have symbol with the same names, it has just happened for the second time.

Similar Threads

  1. Genefile - to generate a test file
    By Sportman in forum Data Compression
    Replies: 2
    Last Post: 17th February 2013, 13:05
  2. Sometimes data look like random... here's an interesting file:
    By Alexander Rhatushnyak in forum The Off-Topic Lounge
    Replies: 29
    Last Post: 25th December 2010, 03:05
  3. An interesting test set
    By nanoflooder in forum Data Compression
    Replies: 12
    Last Post: 13th April 2009, 01:33
  4. Audio test file
    By Mihai Cartoaje in forum Download Area
    Replies: 1
    Last Post: 7th January 2009, 09:47
  5. I found an interesting test set...
    By m^2 in forum Data Compression
    Replies: 7
    Last Post: 3rd January 2009, 14:22

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •