Page 1 of 2 12 LastLast
Results 1 to 30 of 35

Thread: BCM v0.06,0.07 is here! [!]

  1. #1
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts

    Exclamation BCM v0.06,0.07 is here! [!]

    OK, a new release is here!

    What's new:
    • Completely rewritten the BWT part, featuring heavy optimizations and improvements. A very special thanks to Yuta Mori!
    • Added proper allocation order along with /LARGEADDRESSAWARE flag. Now BCM may use a larger blocks (up to -b508751)
    • Some other optimizations and improvements
    Enjoy!

    Update:
    • BCM v0.07 - Fixed a bug in the encoder that may cause the program crash
    Attached Files Attached Files

  2. #2
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Ilia!

  3. #3
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,322
    Thanks
    209
    Thanked 1,001 Times in 526 Posts
    bcm007 -b508751 enwik9 -> 169,427,399
    bcm007 -b488282 enwik9 -> 169,396,680

    Code:
    Z:\BCM>timetest bcm007 -f -b508751 enwik9 enwik9.bcm
    bcm v0.07 by ilia muraviev
    compressing 508751k block...
    compressing 467811k block...
    ratio: 1.355 bpb
    done
    Tested program has wasted 473.985s
    
    Z:\BCM>timetest bcm007 -f -b488282 enwik9 enwik9a.bcm
    bcm v0.07 by ilia muraviev
    compressing 488282k block...
    compressing 488280k block...
    ratio: 1.355 bpb
    done
    Tested program has wasted 472.797s
    
    Z:\BCM>timetest bcm007 -f -d enwik9.bcm 1
    bcm v0.07 by ilia muraviev
    decompressing 508751k block...
    decompressing 467811k block...
    done
    Tested program has wasted 338.875s
    
    Z:\BCM>timetest bcm007 -f -d enwik9a.bcm 1
    bcm v0.07 by ilia muraviev
    decompressing 488282k block...
    decompressing 488280k block...
    done
    Tested program has wasted 341.469s

  4. #4
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    The sorting needs much longer. enwik8 was a few bytes bigger and time 44 against 51 sec

  5. #5
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Check out the decompression time!

  6. #6
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Yes I now tested it on a movie. Also decompression time, because I though you must have improved it there.

    [code]
    >700mb video file

    bcm05 -b20000 715,983,366 bytes 374734ms - 384187ms
    bcm07 715,983,375 bytes 475468ms - 350140ms

    It got slower yes, but decompression speed is much much more important. ~2min slower compression. 0.5min faster decompression could still be counted as an improvement


    But why don't you give such information in the first post instead of writing improvement here, improvement there... .That benefits no one.
    Last edited by Simon Berger; 15th March 2009 at 21:56.

  7. #7
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Note that the decompression time may depend on actual file. Already compressed data is not a good example.

    I always look to the future. New BWT inside BCM is very important step, because it provides some new ways for future compression improvements. It has a different BWT-output. As example, check out the ENWIK9 compression with the same block size. Concluding, new BCM is a new basement for the future BCM versions...

  8. #8
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts

  9. #9
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts

    Very nice Job!

    Very Nice results for this impressive BWT compressor ! Nice improvements in speed ! HI!

  10. #10
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts

    Thumbs up

    Nice !
    BCM might become (if it isn't already) your most interesting compressor !
    Very good ratio/speed : it would be interesting to include the BCM algo in your "main" compressor project...
    Any plans for that ?

    What are the further plans for BCM ?
    Keep on improving BCM

  11. #11
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by pat357 View Post
    Nice !
    BCM might become (if it isn't already) your most interesting compressor !
    Very good ratio/speed : it would be interesting to include the BCM algo in your "main" compressor project...
    Any plans for that ?

    What are the further plans for BCM ?
    Keep on improving BCM
    Yep, I think it's my best compressor to date. I do have plans to make BCM OpenSource as well as add BCM compression method to the PIM archiver.
    But before that moment, BCM should became the best with no doubt. BWMonstr showed better compression, so I think it is possible to reuse some ideas from BWMonstr, if it will not ruin the processing speed. So, the main plans are to improve CM part further, to get <209000 bytes on book1 without preprocessing of any kind. Still, I just started...

  12. #12
    Member
    Join Date
    May 2008
    Location
    CN
    Posts
    45
    Thanks
    0
    Thanked 0 Times in 0 Posts

    compress a text file,bcm only bigger than pim

    690,853 我的团长我的团.7z
    681,586 我的团长我的团.rar
    1,630,579 我的团长我的团.txt
    740,868 我的团长我的团.txt.balz
    633,387 我的团长我的团.txt.bcm
    617,826 我的团长我的团.txt.pim

  13. #13
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    215
    Thanks
    0
    Thanked 0 Times in 0 Posts
    BCM showed great performance on compressing decompressed Lingvo dictionaries, beating a lot of PPMs and absolutely crashing nz -cO.
    Can't wait for it to be a full-size archiver

  14. #14
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    38
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by nanoflooder View Post
    BCM showed great performance on compressing decompressed Lingvo dictionaries, beating a lot of PPMs and absolutely crashing nz -cO.
    Can't wait for it to be a full-size archiver
    Stop, stop, stop. What you mean under decompressed Lingvo dics? More exactly speaking, decompressed with what ?

  15. #15
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    215
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Cool down, mate, that's not what you think.
    I used the source uncompressed .dsl files from which you can create .lsd dictionaries (with DSL Compiler apparently), and measured the compression afterwards.

    Just for the benchmark:

    deu-rus_Lein_and_1_0.dsl:

    .dsl: 60 816 318 bytes

    .lsd: 7 898 537 bytes
    .lsd.7z: 6 164 549 bytes

    .dsl.gz: 6 634 446 bytes
    .dsl.zip: 6 462 062 bytes
    .dsl.rar: 3 854 618 bytes
    .dsl.nz (cO): 3 797 196 bytes
    .dsl.7z (lzma): 3 523 139 bytes
    .dsl.7z (ppm): 2 969 775 bytes
    .dsl.bcm: 2 640 144 bytes

  16. #16
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    38
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by nanoflooder View Post
    Cool down, mate, that's not what you think.
    Blin

  17. #17
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    215
    Thanks
    0
    Thanked 0 Times in 0 Posts
    You're not the only person who's trying to find a way to decompress .lsd

  18. #18
    Member Yuri Grille.'s Avatar
    Join Date
    Mar 2009
    Location
    ****
    Posts
    35
    Thanks
    0
    Thanked 1 Time in 1 Post

    Hi Encode take BCM 0.07 in 50.5 kb

    This is a optimization of the BCM 0.07
    Download >
    [removed]

  19. #19
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    408
    Thanks
    0
    Thanked 5 Times in 5 Posts
    Your post is really confusing. Please exactly point out that this is "only" a compressed executable optimization and no new bcm variant.

  20. #20
    Member Yuri Grille.'s Avatar
    Join Date
    Mar 2009
    Location
    ****
    Posts
    35
    Thanks
    0
    Thanked 1 Time in 1 Post

    Is a compressed executable

    Is a compressed executable with a cleaning process.

    >Your post is really confusing. Please exactly point out that this is "only" a compressed executable optimization and no new bcm variant.

  21. #21
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Yahoo!

    Finally I beated the 210xxx byte barrier for book1 with BCM!

    Currently, the new and upcoming BCM compresses the book1 down to 209xxx bytes!

    The most important is that the CM has about the same complexity! Same complexity meaning about the same or even higher speed! I just added an extra model and done an extreme optimization, including the speed optimizations. To be honest the BCM v0.07 was unoptimized - it uses the same CM as v0.06 having a new BWT part.

    Anyway, the optimization is running and the results becoming better and better...


  22. #22
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    215
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Just today I was like "hmm, there hasn't been a new BCM for a while" - and there you go

  23. #23
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Continue working... My laptop running 24/7 with my optimizer and BCM. Unfortunately, the prediction chain and some other stuff may not be optimized automatically. Now I implement an extremely tricky and smart prediction chain that deals with a regular or so called basic CM model, a tricky sparse model and SSE of course...

    (only 1 liter of Vodka helps here!)

  24. #24
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    38
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by encode View Post
    (only 1 liter of Vodka helps here!)
    Do you prefer Cola or Pepsi ? And what proportions do you use for your coctail ?

  25. #25
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Skymmer View Post
    Do you prefer Cola or Pepsi ? And what proportions do you use for your coctail ?
    Usually, I prefer Coca-Cola, but I like Pepsi as well. Proportions are 50/50. To be honest, usually, I do not drink Vodka. I prefer Heineken beer, and red French wine... Anyway, too much alcohol is bad idea, I prefer to keep my mind clean and clear! I just have too many examples on how people degrade with an alcohol. So, my favorite and everyday drinks are: black/green tea, hibiscus, roibush tea and peppermint tea! I like cola with ice, fruit juices, especially orange! And I drink Kefir! The thing is, Vodka sounds very comical. It's a grotesque. Read the Russian classics, say, Suhovo-Kobilin, the characters drink always a glass of Vodka. And it's funny!

  26. #26
    Member chornobyl's Avatar
    Join Date
    May 2008
    Location
    ua/kiev
    Posts
    153
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I prefer to keep my mind clean and clear!
    Totally agree, i also keep my mind clear for eight years, (no alcohol/coffe/tea) and don't plan to change it

  27. #27
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Question

    Quote Originally Posted by encode View Post
    ...And I drink Kefir!...
    Kefir is a very old Hun drink which sometimes can include less than 1% alcohol due to fermentation of milk. Even "ayran" is "unpopular" all over the world while very popular in Turkiye, I just wonder how did you meet with kefir!? Now, I become to think the other Turkish drinks (şalgam, boza, salep etc) is more popular than I thought!? Am I right?

    Info: The Huns are one of the old Turk tribes in Middle Asia and today's Turkiye is a successor of the Huns.
    BIT Archiver homepage: www.osmanturan.com

  28. #28
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Kefir is nearly national Russian drink! Here in Russia it as famous as a milk... Recently, Ayran became well known in Moscow as well. I think it's due to emigrants and gastarbeiters. Anyway, Turkey is not that far from Russia...
    Russian Federation is a fusion of Europe, Mideast and Asia!

  29. #29
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    38
    Thanked 168 Times in 84 Posts
    Completely agree here. Kefir is just wonderfull drink and very famous in Russia. I think some of us are still remember those minimalistically designed bottles of kefir (and milk too) so wide spreaded in former USSR.
    Ayran... Oh that Ayran! Yes, it came to out country not so long time ago and I don't know how much it loved by people here but at least I know one thing - it almost saved my day one time. I was under heavy hangover a couple years ago and then a friend of mine said to me: "What's up? Feeling sick?". "Yeah" - I said and he recomended me Ayran. What da f**king Ayran I thought but I bought it and drunked it. After 30 mins I have been reborned. Really.

  30. #30
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by encode View Post
    Kefir is nearly national Russian drink! Here in Russia it as famous as a milk... Recently, Ayran became well known in Moscow as well.
    I feel like "a bit shocked" It's a pity that I didn't know popularity of ayran and kefir in Russia.

    Quote Originally Posted by Skymmer View Post
    Completely agree here. Kefir is just wonderfull drink and very famous in Russia. I think some of us are still remember those minimalistically designed bottles of kefir (and milk too) so wide spreaded in former USSR.
    Ayran... Oh that Ayran! Yes, it came to out country not so long time ago and I don't know how much it loved by people here but at least I know one thing - it almost saved my day one time. I was under heavy hangover a couple years ago and then a friend of mine said to me: "What's up? Feeling sick?". "Yeah" - I said and he recomended me Ayran. What da f**king Ayran I thought but I bought it and drunked it. After 30 mins I have been reborned. Really.
    Then try Turkish coffee or hot şalgam I should warn you about one thing: in Turkish definition, "hot" meals & drinks usually refer to "hotter" than Mexican foods So, please careful while drinking
    BIT Archiver homepage: www.osmanturan.com

Page 1 of 2 12 LastLast

Similar Threads

  1. BCM v0.10 is here!
    By encode in forum Data Compression
    Replies: 45
    Last Post: 20th June 2010, 22:39
  2. BCM's future
    By encode in forum Data Compression
    Replies: 17
    Last Post: 9th August 2009, 02:00
  3. BCM v0.05 is here! [!]
    By encode in forum Data Compression
    Replies: 19
    Last Post: 8th March 2009, 22:12
  4. BCM v0.04 is here! [!]
    By encode in forum Data Compression
    Replies: 64
    Last Post: 5th March 2009, 17:07
  5. BCM v0.03 is here! [!]
    By encode in forum Data Compression
    Replies: 25
    Last Post: 14th February 2009, 15:42

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •