BCM v0.06,0.07 is here! [!]

[encode]

OK, a new release is here!

What's new:

• Completely rewritten the BWT part, featuring heavy optimizations and improvements. A very special thanks to Yuta Mori!
• Added proper allocation order along with /LARGEADDRESSAWARE flag. Now BCM may use a larger blocks (up to -b508751)
• Some other optimizations and improvements

Enjoy!

Update:

• BCM v0.07 - Fixed a bug in the encoder that may cause the program crash

[LovePimple]

Thanks Ilia!

[Shelwien]

bcm007 -b508751 enwik9 -> 169,427,399
bcm007 -b488282 enwik9 -> 169,396,680

Z:\BCM>timetest bcm007 -f -b508751 enwik9 enwik9.bcm
bcm v0.07 by ilia muraviev
compressing 508751k block...
compressing 467811k block...
ratio: 1.355 bpb
done
Tested program has wasted 473.985s

Z:\BCM>timetest bcm007 -f -b488282 enwik9 enwik9a.bcm
bcm v0.07 by ilia muraviev
compressing 488282k block...
compressing 488280k block...
ratio: 1.355 bpb
done
Tested program has wasted 472.797s

Z:\BCM>timetest bcm007 -f -d enwik9.bcm 1
bcm v0.07 by ilia muraviev
decompressing 508751k block...
decompressing 467811k block...
done
Tested program has wasted 338.875s

Z:\BCM>timetest bcm007 -f -d enwik9a.bcm 1
bcm v0.07 by ilia muraviev
decompressing 488282k block...
decompressing 488280k block...
done
Tested program has wasted 341.469s

[Simon Berger]

The sorting needs much longer. enwik8 was a few bytes bigger and time 44 against 51 sec

[encode]

Check out the decompression time!

[Simon Berger]

Yes I now tested it on a movie. Also decompression time, because I though you must have improved it there.

[code]
>700mb video file

bcm05 -b20000 715,983,366 bytes 374734ms - 384187ms
bcm07 715,983,375 bytes 475468ms - 350140ms

It got slower yes, but decompression speed is much much more important. ~2min slower compression. 0.5min faster decompression could still be counted as an improvement


But why don't you give such information in the first post instead of writing improvement here, improvement there... .That benefits no one.

[encode]

Note that the decompression time may depend on actual file. Already compressed data is not a good example.

I always look to the future. New BWT inside BCM is very important step, because it provides some new ways for future compression improvements. It has a different BWT-output. As example, check out the ENWIK9 compression with the same block size. Concluding, new BCM is a new basement for the future BCM versions...

[encode]

http://cs.fit.edu/~mmahoney/compression/text.html#1694

[Nania Francesco]

Very Nice results for this impressive BWT compressor ! Nice improvements in speed ! HI!

[pat357]

Nice !
BCM might become (if it isn't already) your most interesting compressor !
Very good ratio/speed : it would be interesting to include the BCM algo in your "main" compressor project...
Any plans for that ?

What are the further plans for BCM ?
Keep on improving BCM

[encode]

Nice !
BCM might become (if it isn't already) your most interesting compressor !
Very good ratio/speed : it would be interesting to include the BCM algo in your "main" compressor project...
Any plans for that ?

What are the further plans for BCM ?
Keep on improving BCM

Yep, I think it's my best compressor to date. I do have plans to make BCM OpenSource as well as add BCM compression method to the PIM archiver.
But before that moment, BCM should became the best with no doubt. BWMonstr showed better compression, so I think it is possible to reuse some ideas from BWMonstr, if it will not ruin the processing speed. So, the main plans are to improve CM part further, to get <209000 bytes on book1 without preprocessing of any kind. Still, I just started...

[l1t]

690,853 我的团长我的团.7z
681,586 我的团长我的团.rar
1,630,579 我的团长我的团.txt
740,868 我的团长我的团.txt.balz
633,387 我的团长我的团.txt.bcm
617,826 我的团长我的团.txt.pim

[nanoflooder]

BCM showed great performance on compressing decompressed Lingvo dictionaries, beating a lot of PPMs and absolutely crashing nz -cO.
Can't wait for it to be a full-size archiver

[Skymmer]

BCM showed great performance on compressing decompressed Lingvo dictionaries, beating a lot of PPMs and absolutely crashing nz -cO.
Can't wait for it to be a full-size archiver

Stop, stop, stop. What you mean under decompressed Lingvo dics? More exactly speaking, decompressed with what ?

[nanoflooder]

Cool down, mate, that's not what you think.
I used the source uncompressed .dsl files from which you can create .lsd dictionaries (with DSL Compiler apparently), and measured the compression afterwards.

Just for the benchmark:

deu-rus_Lein_and_1_0.dsl:

.dsl: 60 816 318 bytes

.lsd: 7 898 537 bytes
.lsd.7z: 6 164 549 bytes

.dsl.gz: 6 634 446 bytes
.dsl.zip: 6 462 062 bytes
.dsl.rar: 3 854 618 bytes
.dsl.nz (cO): 3 797 196 bytes
.dsl.7z (lzma): 3 523 139 bytes
.dsl.7z (ppm): 2 969 775 bytes
.dsl.bcm: 2 640 144 bytes

[Skymmer]

Cool down, mate, that's not what you think.

Blin

[nanoflooder]

You're not the only person who's trying to find a way to decompress .lsd

[Yuri Grille.]

This is a optimization of the BCM 0.07
Download >
[removed]

[Simon Berger]

Your post is really confusing. Please exactly point out that this is "only" a compressed executable optimization and no new bcm variant.

[Yuri Grille.]

Is a compressed executable with a cleaning process.

>Your post is really confusing. Please exactly point out that this is "only" a compressed executable optimization and no new bcm variant.

[encode]

Yahoo!

Finally I beated the 210xxx byte barrier for book1 with BCM!

Currently, the new and upcoming BCM compresses the book1 down to 209xxx bytes!

The most important is that the CM has about the same complexity! Same complexity meaning about the same or even higher speed! I just added an extra model and done an extreme optimization, including the speed optimizations. To be honest the BCM v0.07 was unoptimized - it uses the same CM as v0.06 having a new BWT part.

Anyway, the optimization is running and the results becoming better and better...

[nanoflooder]

Just today I was like "hmm, there hasn't been a new BCM for a while" - and there you go

[encode]

Continue working... My laptop running 24/7 with my optimizer and BCM. Unfortunately, the prediction chain and some other stuff may not be optimized automatically. Now I implement an extremely tricky and smart prediction chain that deals with a regular or so called basic CM model, a tricky sparse model and SSE of course...

(only 1 liter of Vodka helps here!)

[Skymmer]

(only 1 liter of Vodka helps here!)

Do you prefer Cola or Pepsi ? And what proportions do you use for your coctail ?

[encode]

Do you prefer Cola or Pepsi ? And what proportions do you use for your coctail ?

Usually, I prefer Coca-Cola, but I like Pepsi as well. Proportions are 50/50. To be honest, usually, I do not drink Vodka. I prefer Heineken beer, and red French wine... Anyway, too much alcohol is bad idea, I prefer to keep my mind clean and clear! I just have too many examples on how people degrade with an alcohol. So, my favorite and everyday drinks are: black/green tea, hibiscus, roibush tea and peppermint tea! I like cola with ice, fruit juices, especially orange! And I drink Kefir! The thing is, Vodka sounds very comical. It's a grotesque. Read the Russian classics, say, Suhovo-Kobilin, the characters drink always a glass of Vodka. And it's funny!

[chornobyl]

I prefer to keep my mind clean and clear!

Totally agree, i also keep my mind clear for eight years, (no alcohol/coffe/tea) and don't plan to change it

[osmanturan]

...And I drink Kefir!...

Kefir is a very old Hun drink which sometimes can include less than 1% alcohol due to fermentation of milk. Even "ayran" is "unpopular" all over the world while very popular in Turkiye, I just wonder how did you meet with kefir!? Now, I become to think the other Turkish drinks (şalgam, boza, salep etc) is more popular than I thought!? Am I right?

Info: The Huns are one of the old Turk tribes in Middle Asia and today's Turkiye is a successor of the Huns.

[encode]

Kefir is nearly national Russian drink! Here in Russia it as famous as a milk... Recently, Ayran became well known in Moscow as well. I think it's due to emigrants and gastarbeiters. Anyway, Turkey is not that far from Russia...
Russian Federation is a fusion of Europe, Mideast and Asia!

[Skymmer]

Completely agree here. Kefir is just wonderfull drink and very famous in Russia. I think some of us are still remember those minimalistically designed bottles of kefir (and milk too) so wide spreaded in former USSR.
Ayran... Oh that Ayran! Yes, it came to out country not so long time ago and I don't know how much it loved by people here but at least I know one thing - it almost saved my day one time. I was under heavy hangover a couple years ago and then a friend of mine said to me: "What's up? Feeling sick?". "Yeah" - I said and he recomended me Ayran. What da f**king Ayran I thought but I bought it and drunked it. After 30 mins I have been reborned. Really.

[osmanturan]

Kefir is nearly national Russian drink! Here in Russia it as famous as a milk... Recently, Ayran became well known in Moscow as well.

I feel like "a bit shocked" It's a pity that I didn't know popularity of ayran and kefir in Russia.

Completely agree here. Kefir is just wonderfull drink and very famous in Russia. I think some of us are still remember those minimalistically designed bottles of kefir (and milk too) so wide spreaded in former USSR.
Ayran... Oh that Ayran! Yes, it came to out country not so long time ago and I don't know how much it loved by people here but at least I know one thing - it almost saved my day one time. I was under heavy hangover a couple years ago and then a friend of mine said to me: "What's up? Feeling sick?". "Yeah" - I said and he recomended me Ayran. What da f**king Ayran I thought but I bought it and drunked it. After 30 mins I have been reborned. Really.

Then try Turkish coffee or hot şalgam I should warn you about one thing: in Turkish definition, "hot" meals & drinks usually refer to "hotter" than Mexican foods So, please careful while drinking