Results 1 to 11 of 11

Thread: BWCM

  1. #1
    Member
    Join Date
    Nov 2013
    Location
    US
    Posts
    131
    Thanks
    31
    Thanked 29 Times in 19 Posts

    BWCM

    Here is a small test compressor I wrote after reading about bzip2. This uses the Burrows-Wheeler Transform and low-order context modelling. There is also a LZP pre-processing stage to solve string repetitions that slow BWT (aaaa..., ababab..., abcabcabc...) more effectively than RLE in bzip2. It supports blocks from 1 MB to 128 MB.

    Usage:
    BWCM c in out
    BWCM d out in_reconstructed

    Maximum compression:
    BWCM c128 in out
    If this crashes, it means too much memory was used. Use smaller blocks: c64 or c32 instead.
    Attached Files Attached Files
    Last edited by cade; 7th December 2013 at 03:35. Reason: Fixed attached exe

  2. The Following User Says Thank You to cade For This Useful Post:

    Stephan Busch (3rd December 2013)

  3. #2
    Member
    Join Date
    Nov 2013
    Location
    US
    Posts
    131
    Thanks
    31
    Thanked 29 Times in 19 Posts
    If that build doesn't work for you, I have used static linking this time (huge file sizes) so no other libraries are needed, and tested on an old computer running Windows XP. Sorry for the inconvenience.

    Edit: I updated the attached files in the top post.
    Last edited by cade; 3rd December 2013 at 14:08.

  4. #3
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    873
    Thanks
    460
    Thanked 175 Times in 85 Posts
    BWCM crashes when trying to compress the SqueezeChart Audio Testset in .Tar container.

  5. The Following User Says Thank You to Stephan Busch For This Useful Post:

    cade (5th December 2013)

  6. #4
    Member
    Join Date
    Nov 2013
    Location
    US
    Posts
    131
    Thanks
    31
    Thanked 29 Times in 19 Posts
    There was a mistake in my LZP filter. Sorry for the inconvenience; I have fixed it and updated the attachment in the first post.

  7. #5
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    873
    Thanks
    460
    Thanked 175 Times in 85 Posts
    BWCM still crashes on the audio testset.

  8. #6
    Member
    Join Date
    Nov 2013
    Location
    US
    Posts
    131
    Thanks
    31
    Thanked 29 Times in 19 Posts
    Could you please give me a link to download the test set, and tell me if it also crashes the other compressor I uploaded?

    Edit: I found the problem, I assumed lzp_size would need the same number of bits as block size (at most), I forgot that the worst-case for lzp is larger than original (escape codes). Sorry for the inconvenience, I have updated the first post.
    Last edited by cade; 6th December 2013 at 15:12.

  9. #7
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    873
    Thanks
    460
    Thanked 175 Times in 85 Posts
    This version also crashes. This is the output from the cmdline-window:

    E:\TESTSETS>bwcm c128 app.tar app.bw
    BWCM
    Written by Nauful
    Dec 6, 2013
    Block size: 128 MB
    BWCM::Encode: 0.00%
    BWCM::Encode: 39.92%
    BWCM::Encode: 79.85%
    Elapsed: 245887.56ms
    336186368 > 112710722
    E:\TESTSETS>bwcm c128 au.tar au.bw
    BWCM
    Written by Nauful
    Dec 6, 2013
    Block size: 128 MB
    BWCM::Encode: 0.00%
    terminate called after throwing an instance of 'std::bad_alloc'
    what(): std::bad_alloc

    This application has requested the Runtime to terminate it in an unusual way.
    Please contact the application's support team for more information.


    You can download the testset here: http://www.squeezechart.com/TEST_Audio.arc
    The .wav were put in a .tar for this test.

  10. #8
    Member
    Join Date
    Nov 2013
    Location
    US
    Posts
    131
    Thanks
    31
    Thanked 29 Times in 19 Posts
    It used too much memory, try using c64 instead.

    Edit: I fixed the one in the first post to go without LZP if it has a negative gain. Please try that with c128 again.
    Last edited by cade; 7th December 2013 at 03:37.

  11. #9
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    I tested bwcm, cm0, cm0_ext, and cm1 on LTCB. http://mattmahoney.net/dc/text.html#1819

  12. The Following 2 Users Say Thank You to Matt Mahoney For This Useful Post:

    cade (12th December 2013),load (12th December 2013)

  13. #10
    Member
    Join Date
    Nov 2013
    Location
    US
    Posts
    131
    Thanks
    31
    Thanked 29 Times in 19 Posts
    Thanks for testing them. Just a minor nitpick, you spelt my name incorrectly.

    I'll try to get back to developing programs soon. I have some other ideas, but they are more for practical speeds with good compression instead of better compression.

  14. #11
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    Sorry, fixed.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •