BCM v0.03 is here! [!]

[encode]

OK, I have something special for you this time! Please welcome the brand new BCM v0.03! This is a complete rewrite of a program featuring an extremely new and unique Fast CM that designed specially for BWT-output. No more words. Check it out and enjoy!

[encode]

calgary.tar (3,152,896 bytes)
BCM -> 787,078 bytes
BLIZ -> 790,491 bytes
BBB -> 798,705 bytes
MCOMP -> 800,402 bytes

book1 (768,771 bytes)
BCM -> 211,554 bytes
BLIZ -> 212,130 bytes
BBB -> 213,162 bytes
MCOMP -> 217,403 bytes

[m^2]

I'm amazed by your progress.

BOOKSTAR:
BCM 0.03		9262010	~28.5s.
BCM 0.02		9547951	64.610
BBB f 160m		9226430	~128.5
Blizzard c		9590717	21.203
Blizzard c 140000000	9107041	25.641
Blizzard f		9753676	17.719
Blizzard f 140000000	9257193	22.204
YBS  2m			9998527	~21.5
YBS 16m			9444092	~27

[LovePimple]

Thanks Ilia!

[Nania Francesco]

Please Ilia help me , MOC Test failed for BCM in file :
http://veimages.gsfc.nasa.gov/2433/l..._topo_8192.tif conveted to BMP 24B

[Matt Mahoney]

1.0337 in generic test
http://cs.fit.edu/~mmahoney/compression/uiq/

#37 on LTCB
http://cs.fit.edu/~mmahoney/compression/uiq/

LTCB would probably benefit from a larger block size. Compression and speed is pretty good considering the low memory usage.

[Matt Mahoney]

Please Ilia help me , MOC Test failed for BCM in file :
http://veimages.gsfc.nasa.gov/2433/l..._topo_8192.tif conveted to BMP 24B

I tried the original TIF (not converted to BMP) and verified decompression was OK.
27,797,968 -> 25,723,903.

[Nania Francesco]

The problem is in the BMP converted file!

[encode]

The problem is in the BMP converted file!

I'll fix the bug. Probably it's the fake alert in the decoder...

[joerg]

i make a first test on a new machine with Core 2 Duo / 2 GB RAM

The Testfile db.dmp (Oracle-dump-file) has 648331264 bytes.

bcm003 ratio: 0.392 bpb compress to 31783496 bytes 260 s

RINGS v.1.3 (FCM Fast Context Mixing) file compressor. Only for testing
Copyright 2007 by Nania Francesco Antonio (Italy). All rights reserved.

rings13 c compress to 26356464 bytes time= 20.77 s.

RINGS v.1.5c (FCM Fast Context Mixing) file compressor. Only for testing
Copyright 2007-2008 by Nania Francesco Antonio (Italy). All rights reserved.

rings15c c 1 compress to 32986358 bytes time= 23.89 s.
rings15c c 2 compress to 30505178 bytes time= 24.09 s.
rings15c c 3 compress to 28717515 bytes time= 24.28 s.
rings15c c 4 compress to 27461544 bytes time= 24.11 s.
rings15c c 5 compress to 26580793 bytes time= 24.08 s.
rings15c c 6 compress to 25955867 bytes time= 24.03 s.
rings15c c 7 compress to 25649655 bytes time= 24.33 s.
rings15c c 8 compress to 25385543 bytes time= 24.44 s.
rings15c c 9 compress to 25222852 bytes time= 23.83 s.

my result:

bcm performs significantly faster on better hardware
but for now
rings 1.3 and rings 1.5c wins clearly on textfile or oracle-dmp-file
(size and time is lower)

bcm compress better then
rings15c c 1
in the one case - if rings use only low memory

i do not know - may be
bcm would compress better if it use more memory ?

best regards

[encode]

i make a first test on a new machine with Core 2 Duo / 2 GB RAM

The Testfile db.dmp (Oracle-dump-file) has 648331264 bytes.

bcm003 ratio: 0.392 bpb compress to 31783496 bytes 260 s

RINGS v.1.3 (FCM Fast Context Mixing) file compressor. Only for testing
Copyright 2007 by Nania Francesco Antonio (Italy). All rights reserved.

rings13 c compress to 26356464 bytes time= 20.77 s.

RINGS v.1.5c (FCM Fast Context Mixing) file compressor. Only for testing
Copyright 2007-2008 by Nania Francesco Antonio (Italy). All rights reserved.

rings15c c 1 compress to 32986358 bytes time= 23.89 s.
rings15c c 2 compress to 30505178 bytes time= 24.09 s.
rings15c c 3 compress to 28717515 bytes time= 24.28 s.
rings15c c 4 compress to 27461544 bytes time= 24.11 s.
rings15c c 5 compress to 26580793 bytes time= 24.08 s.
rings15c c 6 compress to 25955867 bytes time= 24.03 s.
rings15c c 7 compress to 25649655 bytes time= 24.33 s.
rings15c c 8 compress to 25385543 bytes time= 24.44 s.
rings15c c 9 compress to 25222852 bytes time= 23.83 s.

my result:

bcm performs significantly faster on better hardware
but for now
rings 1.3 and rings 1.5c wins clearly on textfile or oracle-dmp-file
(size and time is lower)

bcm compress better then
rings15c c 1
in the one case - if rings use only low memory

i do not know - may be
bcm would compress better if it use more memory ?

best regards

The catch in BWT's block size. Of course on large text or similar files BWT benefits from a larger block size. BCM has a fixed 32 MB block. Try to compare Rings and BCM on files <=32 MB in size. At the same time a larger block may hurt compression with binary data.

[joerg]

---
The catch in BWT's block size.
Of course on large text files BWT benefits from a larger block size.
BCM has a fixed 32 MB block.
Try to compare Rings and BCM on files <=32 MB in size.
At the same time a larger block may hurt compression with binary data
---
thank you very much for your quickly answer

i will test this

Would it be possible to make a version with a 64 MB block or 128 MB block ?

thank you for your work!

best regards

[encode]

---
The catch in BWT's block size.
Of course on large text files BWT benefits from a larger block size.
BCM has a fixed 32 MB block.
Try to compare Rings and BCM on files <=32 MB in size.
At the same time a larger block may hurt compression with binary data
---
thank you very much for your quickly answer

i will test this

Would it be possible to make a version with a 64 MB block or 128 MB block ?

thank you for your work!

best regards

BCM v0.04 that will be released ASAP (due to a bug fix), will have 64 MB block (~324 MB mem use - still OK). I'm talking about block size importance because you may not beleive how large block may improve compression on files like your one. I'll post some results with various block sizes on ENWIK9 to see the difference a little bit later...

[joerg]

thank you again for your quickly answer

bcm 0.04 sounds very interesting

best regards

[encode]

The bug was fixed. It's indeed a fake alert in the decoder. Since haste makes a waste I'll release a new version tomorrow...

Some testing results on ENWIK9 with different block sizes:

32m -> 192,194,478 bytes
64m -> 185,846,162 bytes
128m -> 180,548,624 bytes
256m -> 175,167,185 bytes

64m block is quite reasonable.

Just tested RINGS v.1.5c on standard files:
book1 -> 226,672 bytes
calgary.tar -> 831,821 bytes
It's no so good...

[m^2]

Why won't you make it configurable?
It will be beneficial for users and for your scores too.
I suggest independent settings for BWT and CM memory.

[joerg]

---
results on ENWIK9 with different block sizes:

32m -> 192,194,478 bytes
64m -> 185,846,162 bytes
128m -> 180,548,624 bytes
256m -> 175,167,185 bytes
---

i think too it would be nice to experiment with several amonts of blocksize

at least it would be wonderful to have a 64mb-
and a (as you has shown better compressing) 256mb- version

if i understand well, then this is the blocksize for bwt - is this right ?
i think - in several cases cm can profit from a bigger blocksize too
but a "step by step" - optimizing is better

best regards

[encode]

Well, for me it's not really interesting to gain compression on weird files like huge pictures and enormous text files like ENWIK using a huge BWT block. As I said, in practice a larger block doesn't mean a better compression. But the main reason is that I do plan to add an auto-block size - i.e. kind of segmentation for higher binary compression...

[m^2]

Well, for me it's not really interesting to gain compression on weird files like huge pictures and enormous text files like ENWIK using a huge BWT block. As I said, in practice a larger block doesn't mean a better compression. But the main reason is that I do plan to add an auto-block size - i.e. kind of segmentation for higher binary compression...

How about solid blocks? They are common. With a lot of text files setting large block size should definitely help. And it's not that artificial example.

Really, nobody's gonna use it on binary files unless by mistake.

[encode]

BCM is a general purpose file compressor. Although it is the best for texts and pictures.

BWT, I even further improved CM compression for next version...

[joerg]

thanks!

--- improved CM compression for next version ---

sounds wonderful !

--- I do plan to add an auto-block size ---

"step by step" ?
i think for now it would be enough to have two versions
one with 64 MB and one with 256 MB blocksize
then we can see the 256 MB-Version is significantly better or not

best regards

[Matt Mahoney]

---
results on ENWIK9 with different block sizes:

32m -> 192,194,478 bytes
64m -> 185,846,162 bytes
128m -> 180,548,624 bytes
256m -> 175,167,185 bytes
best regards

There is a similar table for BBB. For text, there doesn't seem to be any point where using a bigger block size doesn't help.
http://cs.fit.edu/~mmahoney/compression/text.html#1640

The only reason BBB does well on enwik9 is that it uses a single 1 GB block size with some tricks so it runs with 1.25 GB memory.

[Hahobas]

Here's a little test on my Athlon 64 3000+ (single core) machine with BCM.

bcm003 enwik8 - 22,007,655 bytes ( Compress: 69sec, Decompress 61sec)
bcm002a enwik8 - 23,761,415 ( Compress: 138sec, Decompress: 56 sec )

Great progress! Can't wait for the next version.

[encode]

To next version I already added an optimized SSE with linear interpolation at last.
BCM have interesting CM consisting of two parts:
1. Set of models with static mixing with not just order-0, but with special order-1 and order-2
2. SSE model (which may be considered as an additional order-0 model)

All these stuff provides one of the best results of BWT-output coding so far! Especially, new BCM v0.04. It closely beats PPMd on book1!

[easy]

наша фирма заинтересована в использование этого компрессора в программирование видео игр для приставок Nintendo DS, iPhone, Wii

1) заинтересованы ли вы в том, чтобы мы использовали ваш комппессор?
2) возможно ли декомпрессировать етим компрессором формат BCM с процессором 66Mhz и небольшой RAM?

easy

[encode]

наша фирма заинтересована в использование этого компрессора в программирование видео игр для приставок Nintendo DS, iPhone, Wii

1) заинтересованы ли вы в том, чтобы мы использовали ваш комппессор?
2) возможно ли декомпрессировать етим компрессором формат BCM с процессором 66Mhz и небольшой RAM?

easy

Компрессор BCM имеет опции которые можно настроить для нужной производительности и с учетом конкретного оборудования. Например при малом оъеме доступной оперативной памяти, возможно использование более малого блока для сжатия. Для более слабого процессора можно "облегчить" контекстную модель - например убрав некоторые составляющие части этой модели как то SSE/APM, интерполяцию, отдельные счетчики. В этом плане BCM очень гибок.
Я заинтересован и готов к сотрудничеству!
Также со мной можно связаться через Private Messages на этом форуме или через ICQ - номер моей аськи указан у меня в профиле!