BWCM

[cade]

Here is a small test compressor I wrote after reading about bzip2. This uses the Burrows-Wheeler Transform and low-order context modelling. There is also a LZP pre-processing stage to solve string repetitions that slow BWT (aaaa..., ababab..., abcabcabc...) more effectively than RLE in bzip2. It supports blocks from 1 MB to 128 MB.

Usage:
BWCM c in out
BWCM d out in_reconstructed

Maximum compression:
BWCM c128 in out
If this crashes, it means too much memory was used. Use smaller blocks: c64 or c32 instead.

[cade]

If that build doesn't work for you, I have used static linking this time (huge file sizes) so no other libraries are needed, and tested on an old computer running Windows XP. Sorry for the inconvenience.

Edit: I updated the attached files in the top post.

[Stephan Busch]

BWCM crashes when trying to compress the SqueezeChart Audio Testset in .Tar container.

[cade]

There was a mistake in my LZP filter. Sorry for the inconvenience; I have fixed it and updated the attachment in the first post.

[Stephan Busch]

BWCM still crashes on the audio testset.

[cade]

Could you please give me a link to download the test set, and tell me if it also crashes the other compressor I uploaded?

Edit: I found the problem, I assumed lzp_size would need the same number of bits as block size (at most), I forgot that the worst-case for lzp is larger than original (escape codes). Sorry for the inconvenience, I have updated the first post.

[Stephan Busch]

This version also crashes. This is the output from the cmdline-window:

E:\TESTSETS>bwcm c128 app.tar app.bw

BWCM

Written by Nauful

Dec 6, 2013

Block size: 128 MB

BWCM::Encode: 0.00%

BWCM::Encode: 39.92%

BWCM::Encode: 79.85%

Elapsed: 245887.56ms

336186368 > 112710722

E:\TESTSETS>bwcm c128 au.tar au.bw

BWCM

Written by Nauful

Dec 6, 2013

Block size: 128 MB

BWCM::Encode: 0.00%

terminate called after throwing an instance of 'std::bad_alloc'

  what():  std::bad_alloc



This application has requested the Runtime to terminate it in an unusual way.

Please contact the application's support team for more information.

You can download the testset here: http://www.squeezechart.com/TEST_Audio.arc
The .wav were put in a .tar for this test.

[cade]

It used too much memory, try using c64 instead.

Edit: I fixed the one in the first post to go without LZP if it has a negative gain. Please try that with c128 again.

[Matt Mahoney]

I tested bwcm, cm0, cm0_ext, and cm1 on LTCB. http://mattmahoney.net/dc/text.html#1819

[cade]

Thanks for testing them. Just a minor nitpick, you spelt my name incorrectly.

I'll try to get back to developing programs soon. I have some other ideas, but they are more for practical speeds with good compression instead of better compression.

[Matt Mahoney]

Sorry, fixed.