Results 1 to 1 of 1

Thread: Testing compressors with artificial data

  1. #1
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts

    Testing compressors with artificial data

    I made up a test corpus of artificial data files and tested it on a few popular compressors (attached). I think it makes sense to test any new compression algorithm on simple cases, such as a file of all zero bytes, or all random bytes, or alternating zero and random bytes. I found a surprising number of simple cases that can break a lot of compressors (as in compressing poorly). For example, a lot of compressors will not be able to compress (at least not well) the simple case of counting 2 or 4 byte integers (0, 1, 2, 3...). I found a lot of other cases that broke zpaq, which should give me something to work on.

    The attached zip has a readme.txt that describes how the data files were created, and some test results with several compressors and my comments. There is also a small (5 KB) zpaq archive inside which contains the test files, or you can use the scripts (in Windows) to generate them or variations of them. The largest file is 100 MB containing 2 copies of a 50 MB pseudo-random string. I was able to make the archive small by using ZPAQL config files to generate all of the test files, then using the same config files to build the archive. Otherwise the archive would be at least 50 MB.
    Attached Files Attached Files

Similar Threads

  1. Some of my toy compressors
    By RichSelian in forum Data Compression
    Replies: 27
    Last Post: 6th October 2011, 06:09
  2. What's wrong with my testing script?
    By m^2 in forum Data Compression
    Replies: 20
    Last Post: 21st September 2008, 20:24
  3. Help beta testing QuickLZ 1.40 with the new test framework
    By Lasse Reinhold in forum Forum Archive
    Replies: 10
    Last Post: 19th April 2008, 17:16
  4. image compressors
    By maadjordan in forum Forum Archive
    Replies: 5
    Last Post: 13th August 2007, 10:28
  5. Fastest Compressors
    By LovePimple in forum Forum Archive
    Replies: 0
    Last Post: 1st November 2006, 06:36

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •