BCM v0.09 - The ultimate BWT-based file compressor!

[encode]

Let me introduce the best piece of my software! Please welcome, the ultimate BWT-based file compressor - BCM v0.09! This is kind of a final release, since I have no spare time completely...



Anyway, BCM v0.09 on some test files:

book1 -> 208,726 bytes
calgary.tar -> 775,271 bytes
ENWIK5 -> 1,195,166 bytes
ENWIK8 -> 20,625,697 bytes
world95.txt -> 463,725 bytes
fp.log -> 551,378 bytes
bible.txt -> 721,591 bytes
3200.txt-> 3,641,305 bytes

[PiPPoNe92]

yessssss!!! thanks encode!!! you're great!! improve compression ratio than v0.08!!

[LovePimple]

Thanks Ilia!

[Skymmer]

My congrats with new BCM, Ilia
Some tests:

              | encode time |     size    | decode time |
--------------------------------------------------------|
              |         PAK0.PAK (302 895 669)          |
--------------------------------------------------------|
0.07 -b65536  |  181.813    | 117 053 060 |  135.813    |
0.08 e64      |  227.422    | 116 587 092 |  185.234    |
0.09 -b64     |  383.133    | 115 564 822 |  343.401    |
--------------------------------------------------------|
              |          ENWIK8 (you know :) )          |
--------------------------------------------------------|
0.07 -b102400 |   61.265    |  20 770 673 |   45.109    |
0.08 e100     |   77.828    |  20 744 613 |   61.172    |
0.09 -b100    |  110.964    |  20 625 697 |   95.135    |
--------------------------------------------------------|
              |         NKOEF.DBF (228 819 632)         |
--------------------------------------------------------|
0.07 -b32768  |  108.078    |   4 432 347 |   89.859    |
0.08 e32      |  154.953    |   4 393 106 |  131.453    |
0.09 -b32     |  214.163    |   4 245 588 |  192.951    |
--------------------------------------------------------'

By the way, on ENWIK8 BWT_reorder_v2 + BCM 0.09 -b100 = 20 555 272

[PiPPoNe92]

encode, bcm is more powerful than 7z, but it takes a few ram and cpu, if you do bcm 0.1, can you improve compression increase cpu and ram?? because now general pc have 4 gb ram and 2 cpu thanks for answers

[encode]

Use '-b' option! As example:
bcm -b320 enwik9
bcm -b200 pak0.pak
bcm -b471 textures.tar

[PiPPoNe92]

don't change nothing! the compression ratio and final size are equals

[encode]

Well, I might be if the block size (-b) is equal or bigger than the file size...

As example, if you about to compress 4 MB file

bcm file
bcm -b5 file
bcm -b500 file

will produce the same results.

But if the file size is 800 MB, the -b option will take a serious effect.

But note, in some cases like with EXE-files, TAR-archive with mixed stuff, etc. a smaller block might be better:

bcm -b5 reaktor.exe

[encode]

Looks like the faster BCM engine (v0.07,v0.0 was more successful and popular. I'm just thinking about to release a new BCM v0.10 with much faster/lighter compression engine. Incorporating some ideas and optimizations from the latest v0.09 and simplicity of v0.07. I think it will be even more simpler than v0.07...

[encode]

The brand new BCM v0.10 should be the fastest BCM ever. I've done many optimizations and time measurements using RDTSC to find out the lightest and yet efficient CM back end. 209xxx on book1 it has less complex CM part, new CM is even simpler than that BCM v0.07 had. On some files it may provide a slightly worse compression compared to the v0.07, on some may not - providing a higher compression. Overall, it's something like BCM v0.07, but notable faster - due to the simplest CM back-end and some new stuff that I've invented working on BCM v0.09. I think new BCM is very interesting piece of software, firstly, at the practical point of view...

[PiPPoNe92]

thanks encode for the info!!

[encode]

Here are some of the testing results of a test version of the new BCM v0.10. As a note new BCM is faster than v0.07 and much faster than any newer...

book1 -> 209,781 bytes
bible.txt -> 726,154 bytes
3200.txt -> 3,663,617 bytes
fp.log -> 561,549 bytes
...

[zody]

Sounds VERY great; perfect efficiency

[encode]

Additionally, new BCM v0.10 more likely will be like a full-blown command-line compressor. With an extra error checking, more stable&efficient code, own file format with a small header, etc. I did many optimizations not only that refers to a main compression engine, as example a faster unbwt (decompression), but CLI-related stuff as well. Faster percentage calculation, more stable command-line parsing&file name generation, etc. In other words, it's the entire program rewrite! I really checked and rechecked each line of code. The release is soon...

[squxe]

Encode, what's about idea of adding new BCM into your Pim Archiver? I think, it will be very interesting

[encode]

Encode, what's about idea of adding new BCM into your Pim Archiver? I think, it will be very interesting

BCM is still an experimental file compressor - since it's under heavy development and some compression related changes may occur, breaking any compatibility. I do plan to add BCM to PIM archiver as one of the PIM's algorithms and/or as .BCM file extraction. Anyway, the main goal of the BCM is a freeware file compression tool and a commercial data compression library; i.e. it will never be an open source as my other projects like BALZ or QUAD. Although, I do plan to release binaries for Linux/Unix.

[encode]

Just made a 64-bit version of the BCM. And it runs faster than its 32-bit version. So, the main version will be for 64-bit platforms. Alternatively, the BCM package will contain the X86 compile.

[Vacon]

Hello everyone,

Just made a 64-bit version of the BCM. And it runs faster than its 32-bit version. So, the main version will be for 64-bit platforms. Alternatively, the BCM package will contain the X86 compile.

So the children will get something to play with soon?

Best regards!

[encode]

Actually, the new BCM already written some time ago. I'm just playing with some optimizations and thinking about more correct error messages.

[encode]

New BCM was developed and optimized under at least three followed processors:
Pentium 4
Core 2 Duo
Core 2 Quad

I think I'll look more to the 64-bit side. I optimized some parts of the BCM specially for the 64-bit machines. At some point, I'm about to release kind of a final, practical BCM, to keep compatibility between versions. Some ways of the optimizations are incompatible with each other. That's why I need to choose a right way. Testing under all these processors helped me a lot.

[encode]

New BCM v0.10 will be released very soon. It's completely written for now, I just need a more testing with the max block sizes. Anyway, the key differences:

• New BCM displays progress for the entire file, not just a current block
• For the first time, of all of my command line compressors, new BCM has it's own file format, with its own file header, file ID, etc.
• Better error handling and CL user interface
• And of course, the brand new and much more improved BCM's compression engine
• Concluding, it's a complete data compression tool with its non-experimental look and feel

After releasing this piece of software, I'll continue working on my byte-oriented PPMX...

[Fu Siyuan]

Highly expecting!

[Senoble]

any changes open sourcing this project?

EDIT: Oh, I just find it. Thanks. https://sourceforge.net/projects/bcm/