BCM v0.04 is here! [!]

[m^2]

Just to backup my words:
Look at metacompressor and text tests big enough to get quite reliable decompression speed measurements. There is always (FreeArc) LZP+PPMD with some parameters that's stronger and decompresses faster than BCM.
Sometimes (File3:log) the difference is really huge.

[encode]

I have an idea about adding compression modes:
9 - 320 MB block
8 - 160 MB block
7 - 80 MB block
6 - 40 MB block
5 - 20 MB block
4 - 10 MB block
3 - 5 MB block
2 - 2.5 MB block
1 - 1.25 MB block



Along with an improved CM, this might be really cool!

[osmanturan]

Are we really need such a big block size (>64 MiB) for any practical purposes? Or do you target some benchmarks for beating the other competitor with more memory? For me, 64 MiB at max is enough. So, memory requirements will be 5n=5x64=320+your context model. It's enough big already.

[encode]

Are we really need such a big block size (>64 MiB) for any practical purposes? Or do you target some benchmarks for beating the other competitor with more memory? For me, 64 MiB at max is enough. So, memory requirements will be 5n=5x64=320+your context model. It's enough big already.

Or even better I'll add an option of a block size in MB. Yep, why Blizzard can be tested with 333 MB block and BCM not?
Anyway, you may set a block size to any number you want.

[encode]

BTW, even new context model is small enough - ~700 KB

[encode]

BCM v0.05, option 9:

ENWIK9 -> 172,180,796 bytes

[joerg]

encode wrote:
---
Anyway, you may set a block size to any number you want.
---

sounds wonderful

we are ready to test

[nanoflooder]

BCM v0.05, option 9:

ENWIK9 -> 172,180,796 bytes

Wow One of the best BWT to date?.. nz -cO is your only competitor I don't take BBB cause it's waaay too slow for r/l

Good job!

[encode]

Wow One of the best BWT to date?.. nz -cO is your only competitor I don't take BBB cause it's waaay too slow for r/l

Good job!

BCM with 1GB block will beat BBB! But we need a 64-bit system with more than 5 GB of RAM installed. And NZ may beat BCM due to its text preprocessing only, AFAIK.
Here, option 9 means 320 MB block. Also I'll test it, say, with 350 MB one.

[encode]

BTW, CCMX and even CMM4 are beaten as well...

[Bulat Ziganshin]

why don't compile 64-bit version? i sure there are people here with 16gb of ram

[encode]

why don't compile 64-bit version? i sure there are people here with 16gb of ram

No problem! But at the moment I'm busy optimizing new CM - it's not really a trivial task and takes a huge amount of time...
If someone have at least 6 GB of RAM installed, please let me know...

[joerg]

i have a windows 2003 server 32bit
and a win xp 32 bit with 4 GByte RAM
so i can give full 2 GByte to the application
- want to test

[encode]

i have a windows 2003 server 32bit
and a win xp 32 bit with 4 GByte RAM
so i can give full 2 GByte to the application
- want to test

New BCM will have no block size limit.

usage: bcm e[nn]|d in out
or
usage: bcm e[n]|d in out
or
usage: bcm e[#]|d in out

Vote for the default block size! I think it should be about 5 MB.

[osmanturan]

My homemate has a Q6600 with 6 GiB RAM But, I won't be in my flat until 2 March. Maybe you can finish the other improvements until that date.

[m^2]

New BCM will have no block size limit.

usage: bcm e[nn]|d in out

Vote for the default block size! I think it should be about 5 MB.

120 MB?

[encode]

120 MB?

120*5=600 MB - too heavy. 8/16 MB block can be better. An older BWT compressors have 4/5 MB block, taking into account modern hardware we may use a larger default block.

[encode]

mcomp has a 64MB block by default... good idea!

[m^2]

120*5=600 MB - too heavy. 8/16 MB block can be better. An older BWT compressors have 4/5 MB block, taking into account modern hardware we may use a larger default block.

That's exactly how I calculated.
Come on, it's experimental compressor, compatibility with 98% of computers is enough.

[encode]

That's exactly how I calculated.
Come on, it's experimental compressor, compatibility with 98% of computers is enough.

Yep, maybe you're right!

Anyway, the user may set any block size in MB, e.g.

bcm e400 enwik9 enwik9.bcm

[encode]

Or maybe add a mem option - i.e. mem usage.

say

bcm e1700 enwik9 enwik9.bcm

block size = mem/5

NanoZip like approach. This can be more useful. What do you think?

[joerg]

i think

bcm e nnnn source target.bcm

will be nice

ask: which amount of memory is the minimum to work?

5 MByte - 1 MByte blocksize ??

for test purposes it would be wonderful
if the program can at start write to the console
the amount of available memory
and the amount of used memory

as default i propose

nnnn = 0960 - 960 MByte - 192 MByte blocksize

if there is not enough memory available, then
maybe do a automatic fallback to

nnnn = 0320 - 320 MByte - 64 MByte blocksize

if this amount of memory is too not available, then
maybe do a automatic fallback to

nnnn = 0040 - 40 MByte - 8 MByte blocksize
...
nnnn = 0005 - 5 MByte - 1 MByte blocksize

best regards

[encode]

ask: which amount of memory is the minimum to work?

Actually, I wrote a special BCM lite - which uses 193 KB of memory, compressing better than ZIP.

5 MByte - 1 MByte blocksize ??

With mem option - yes. But I still prefer the block size in MB...

[encode]

BCM with various block sizes on ENWIK9:

64m -> 185,309,765 bytes
120m -> 180,155,865 bytes
300m -> 173,440,311 bytes
310m -> 172,934,216 bytes
340m -> 172,219,709 bytes
333m -> 172,214,743 bytes
350m -> 172,209,734 bytes
360m -> 172,185,320 bytes
320m -> 172,180,796 bytes
370m -> 172,125,800 bytes
375m -> 172,091,083 bytes

[Black_Fox]

Tested, 13,616,873 0.03 failed decompression at one file and hasn't been listed.

[encode]

Tested, 13,616,873

Thanks a lot!

0.03 failed decompression at one file and hasn't been listed.

It's thread about v0.04!

[Nania Francesco]

MOC Test passed for last version ! Very good results !

[encode]

Actually, new BCM's compression engine is done a long time ago. Just currently I have too little spare time...
Anyway, finally, BCM will have a proper GZIP-like command-line interface. Strangely, if I worked so hard on it's compression engine, running optimizer for a few weeks and so, and at the same time kept such dummy command-line interface.

bcm [options] input [output]

As expected, -d means decode/decompress. -f means force overwrite of output file. The -b option sets the block size. Not decided yet about, should user specify the block size in KB or MB or in both [k,m]. Also not decided yet about the E8/E9 filter - should I remove it or add a new filter that will work with a large blocks...

[joerg]

if you think the blocksize as parameter is better - why not
then
i propose to specify blocksize in KiloBytes

for example -b393216
means blocksize = 393216 KB = 384 MB .. using 1920 MByte RAM

but wishes

1. a option -V = verbose
would be wonderful for test purposes
"program writes at start to the console
the amount of available memory
and the amount of used memory"

2. if blocksize option is not selected
then i propose
default blocksize = 192 MByte .. using 960 MByte RAM

with an automatic fallback to a lower blocksize
if there is not enough memory available

blocksize = 65536 KB .. using 320 MByte RAM

if this amount of memory is too not available,
then maybe do a automatic fallback to

blocksize = 8192 KB .. using 40 MByte RAM
..
blocksize = 1024 KB .. using 5 MByte RAM

3. if you have the time it would be very nice to have a

option -r
(recursivly compress a whole directory with subdirecty-tree)

bcm -r input output

where input is a directory like D:\PICTURES\

4. have a option -nofilter
to switch off the E8/E9 filter

best regards

[chornobyl]

I would prefer block size in MEGAbytes (not much people will use blocks under 1 MB imho).