Page 1 of 2 12 LastLast
Results 1 to 30 of 46

Thread: BCM v0.10 is here!

  1. #1
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts

    Cool BCM v0.10 is here!

    BCM is a high performance BWT-based file compressor.

    The brand new version is kind of a final. At some point new BCMs compression engine is mature enough to keep it as is without any modifications and thus breaking compatibility. Generally speaking, this version is some kind of a modern BZIP2 with much more stronger (book1 -> 209,794 bytes) and yet efficient compression. New BCM has it's own file header, file identification and data integrity checking. As expected, it supports large files (64-bit) to compress even extremely large amount of data.

    To obtain the fastest compression:
    bcm -b1 somefile
    or even
    bcm -b2 somefile

    For the best text compression, use the largest block possible:
    bcm -b512 somefile

    Memory usage is about BlockSize*5. Thus with 512 MB block you will need 2560 MB of free RAM. If you have no such amount of free memory, try a smaller block.

    With the binary compression, "larger block=better compression" rule not really works. For some files 8 MB block might be optimum for others not, try to experiment.

    To decompress:
    bcm -d somefile.bcm

    Just as with GZIP/BZIP2, as expected.

    Anyway, enjoy!

    bcm010.zip - x64 executable - the main version
    bcm010-x86.zip - x86 executable - kept mostly for compatibility with old CPUs

    Attached Files Attached Files

  2. #2
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Ilia!

  3. #3
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    38
    Thanked 168 Times in 84 Posts
    Nice! Ilia, thanks Here is updated result table.

    Code:
                  | encode time |     size    | decode time |
    --------------------------------------------------------|
                  |         PAK0.PAK (302 895 669)          |
    --------------------------------------------------------|
    0.07 -b65536  |  181.813    | 117 053 060 |  135.813    |
    0.08 e64      |  227.422    | 116 587 092 |  185.234    |
    0.09 -b64     |  383.133    | 115 564 822 |  343.401    |
    0.10 -b64     |  142.834    | 117 956 984 |  102.460    |
    --------------------------------------------------------|
                  |          ENWIK8 (you know :) )          |
    --------------------------------------------------------|
    0.07 -b102400 |   61.265    |  20 770 673 |   45.109    |
    0.08 e100     |   77.828    |  20 744 613 |   61.172    |
    0.09 -b100    |  110.964    |  20 625 697 |   95.135    |
    0.10 -b100    |   57.005    |  20 811 710 |   38.097    |
    --------------------------------------------------------|
                  |         NKOEF.DBF (214 711 184)         |
    --------------------------------------------------------|
    0.07 -b32768  |  100.810    |   4 320 824 |   84.196    |
    0.08 e32      |  145.739    |   4 277 428 |  123.581    |
    0.09 -b32     |  200.461    |   4 140 464 |  180.572    |
    0.10 -b32     |   89.213    |   4 529 380 |   56.010    |
    --------------------------------------------------------'
    EDIT: Tested on AMD64 4000+ (Single Core), Win XP Pro SP2 (32 bit)
    Last edited by Skymmer; 12th December 2009 at 19:32.

  4. #4
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Machine specs? (On my Core2Duo/Core2Quad with 4 GB RAM machines and under Windows 7 new BCM runs notable faster)

  5. #5
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    863
    Thanks
    459
    Thanked 257 Times in 105 Posts
    BCM -b100 Enwik8 --> 20.81% in 39s
    BCM -b256 Enwik9 --> 17.52% in 423s

    Machine specs : core 2 duo E8400 @ 3GHz, 2GB RAM, Windows Seven x32

    Nice work Ilia !
    Last edited by Cyan; 12th December 2009 at 19:55.

  6. #6
    Member zody's Avatar
    Join Date
    Aug 2009
    Location
    Germany
    Posts
    90
    Thanks
    0
    Thanked 1 Time in 1 Post
    Great Work

    bcm -b512 enwik9 16,9871532 % in 362 sec

    W7 x64; e8200@3,6ghz; 4gb 1066 ram

  7. #7
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    38
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by encode View Post
    Machine specs? (On my Core2Duo/Core2Quad with 4 GB RAM machines and under Windows 7 new BCM runs notable faster)
    Updated original message.

  8. #8
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,474
    Thanks
    26
    Thanked 121 Times in 95 Posts
    What about Linux version?

  9. #9
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Linux version needs some extra work. So, if BCM will be popular enough - I'll do it!

    Please note, BCM v 0.10-x86 is not BCM v0.10 - they are different! So, please post the correct version info! Or else I will be forced to keep just one 64-bit version! New BCM is designed for 64-bit CPUs...

  10. #10
    Programmer Gribok's Avatar
    Join Date
    Apr 2007
    Location
    USA
    Posts
    159
    Thanks
    0
    Thanked 1 Time in 1 Post
    Tested on windows Server 2008 R2 with Intel Q9400 on different block sizes. CPU Utilization is only 25%. Bug.
    Enjoy coding, enjoy life!

  11. #11
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Gribok View Post
    Tested on windows Server 2008 R2 with Intel Q9400 on different block sizes. CPU Utilization is only 25%. Bug.
    Intel? Core?2 Quad Processor Q9400 has four cores. BCM is a single-threaded app, thus it uses just one core (25%)

  12. #12
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    38
    Thanked 168 Times in 84 Posts
    Quote Originally Posted by encode View Post
    Please note, BCM v 0.10-x86 is not BCM v0.10 - they are different!
    Surely they're different but... what exactly do you mean? The maximum block size (as you said) is 512 then memory requirement is 2560 MB, which is, I suppose, available with 32bit OS. So, what then? Speed? Something else?

    Quote Originally Posted by encode View Post
    So, please post the correct version info! Or else I will be forced to keep just one 64-bit version! New BCM is designed for 64-bit CPUs...
    I'm really curious why you're so highly against the 32bit OSes.

  13. #13
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Skymmer View Post
    Surely they're different but... what exactly do you mean? The maximum block size (as you said) is 512 then memory requirement is 2560 MB, which is, I suppose, available with 32bit OS. So, what then? Speed? Something else?


    I'm really curious why you're so highly against the 32bit OSes.
    Additional instructions! Like I said, extra registers and native 64-bit processing make the difference.
    I'm not really against it, but 64-bit platforms gives you a little bit more processing power... I saw the assembler listings of 32/64-bit versions, and I really liked that's going on with 64-bit one.

    Both BCMs are different, it's not just different compiles. They have a different code inside. Additionally, I was focused on 64-bit version only... (Just before the release I optimized the 32-bit version as well, though)

  14. #14
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    863
    Thanks
    459
    Thanked 257 Times in 105 Posts
    Please note, BCM v 0.10-x86 is not BCM v0.10 - they are different! So, please post the correct version info!
    Well, in this case, maybe using a different version number or a different name would help lessening confusion...

  15. #15
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Cyan View Post
    Well, in this case, maybe using a different version number or a different name would help lessening confusion...
    Look at both programs' banner - v0.10-x86 is not v0.10! In fact, it's two different version numbers!

  16. #16
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    863
    Thanks
    459
    Thanked 257 Times in 105 Posts
    Ilia, you know what i mean...

    Using "binary-x86-version" and "binary-x64-version" is by convention referring to 2 identicals programs using 2 different compilation schemes.
    This can lead to different performances, right, but no one actually expects both programs to have different code/algorithm within.
    Last edited by Cyan; 15th December 2009 at 00:48.

  17. #17
    Member
    Join Date
    Dec 2009
    Location
    California
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    As long as the generated bitstreams are compatible it shouldn't make a difference. I'll test on vista x64 when I get home! Just to be sure they're compatible, I'll test each by decoding the other's encode.

  18. #18
    Member
    Join Date
    Jun 2008
    Location
    USA
    Posts
    111
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I don't have 64-bit Windows, but it would interesting to see benchmarks of both "versions" under e.g. Win7 x64.

  19. #19
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    It seems compression is a little worse than bcm 0.09 although about twice as fast. But compare with 0.05.

    bcm 0.10 x86 -b370 enwik9 -> 172,570,245, 758 sec
    bcm 0.10 x86 -b328 enwik9 -> 172,653,295, 761 sec
    bcm 0.05 -b327680 enwik9 -> 172,180,796, 684 sec

    http://mattmahoney.net/dc/text.html#1686

  20. #20
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Matt Mahoney View Post
    It seems compression is a little worse than bcm 0.09 although about twice as fast. But compare with 0.05.

    bcm 0.10 x86 -b370 enwik9 -> 172,570,245, 758 sec
    bcm 0.10 x86 -b328 enwik9 -> 172,653,295, 761 sec
    bcm 0.05 -b327680 enwik9 -> 172,180,796, 684 sec

    http://mattmahoney.net/dc/text.html#1686
    Look at the decompression speed!

  21. #21
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Additionally, at ENWIK9, block size must be 477 MB (i.e. the half of the file)

  22. #22
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Tested under my junior machine: Core2Duo@2.40GHz, DDR2-800 4GB RAM, Windows7 x64

    bcm -b5 enwik5
    5000000 -> 1205884 in 1 sec
    1205884 -> 5000000 in 1 sec

    bcm -b100 enwik8
    100000000 -> 20811710 in 45 sec
    20811710 -> 100000000 in 31 sec

    bcm -b477 enwik9
    1000000000 -> 169843006 in 522 sec
    169843006 -> 1000000000 in 353 sec

    I need a faster RAM...

  23. #23
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    Yes, decompression is faster.
    bcm 0.05, 535 sec.
    bcm 0.10 x86, 483 sec.

    Unfortunately I have 32 bit Vista so I can't test with a block size larger than 370 MB.

  24. #24
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Actually, the 32-bit version was compiled with the /LARGEADDRESSAWARE switch, thus you may use 512 MB blocks under 32-bit Windows...

  25. #25
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,501
    Thanks
    741
    Thanked 664 Times in 358 Posts
    Quote Originally Posted by encode View Post
    Actually, the 32-bit version was compiled with the /LARGEADDRESSAWARE switch, thus you may use 512 MB blocks under 32-bit Windows...
    32-bit windows provides only 2gb of memory to programs. one should use 64-bit windows or /3gb trick

  26. #26
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    Yes, I tried larger blocks and decreased 10 MB at a time until it would run. My computer has 3 GB but only 2 GB available to each process.

  27. #27
    Programmer michael maniscalco's Avatar
    Join Date
    Apr 2007
    Location
    Boston, Massachusetts, USA
    Posts
    113
    Thanks
    11
    Thanked 88 Times in 26 Posts
    Quote Originally Posted by Matt Mahoney View Post
    Yes, I tried larger blocks and decreased 10 MB at a time until it would run. My computer has 3 GB but only 2 GB available to each process.
    Ilia, perhaps it would be useful if you added a switch which selects the largest blocksize available on for the given machine. It might save testers a bit of time.

  28. #28
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by michael maniscalco View Post
    Ilia, perhaps it would be useful if you added a switch which selects the largest blocksize available on for the given machine. It might save testers a bit of time.
    Maybe it's not that good idea. Since with the same options BCM will provide different compression results on different machines...

    Quote Originally Posted by Matt Mahoney
    Yes, I tried larger blocks and decreased 10 MB at a time until it would run. My computer has 3 GB but only 2 GB available to each process.
    Enabling the /3GB mode in Windows Vista.

    If you get a "not enough memory", "out of memory", "low memory" or a similar type of RAM error in a Windows Vista application... Windows Vista no longer uses the BOOT.INI file, so there is a different method for setting the "3GB" switch which enables to use 3GB of RAM for applications in 32-bit Windows (otherwise they use just 2GB - maximum).
    If you want to make accessible 3GB RAM for your CAD applications in Windows Vista, use the BCDedit.exe tool (Boot Configuration Data Editor). The 3GB memory mode can be enabled with the command:
    bcdedit /set IncreaseUserVa 3072
    and disabled by the command:bcdedit /deletevalue IncreaseUserVa
    Run this command from a command window with Administrator priviledge - i.e. (in the Start menu) type CMD and press Ctrl+Shift+Enter, or select "Command Prompt" (Accessories), right-click on it and choose "Run as Administrator". Restart your PC after this change.
    P.S.
    BTW, Matt you may add BCM's compression results under my machine listed above.

  29. #29
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	bcm.png 
Views:	682 
Size:	18.6 KB 
ID:	1172  

  30. #30
    Member Fu Siyuan's Avatar
    Join Date
    Apr 2009
    Location
    Mountain View, CA, US
    Posts
    176
    Thanks
    10
    Thanked 17 Times in 2 Posts
    Quote Originally Posted by encode View Post
    What does it mean? Do you plan to integrate BCM into PIM?

Page 1 of 2 12 LastLast

Similar Threads

  1. BCM's future
    By encode in forum Data Compression
    Replies: 17
    Last Post: 9th August 2009, 02:00
  2. BCM v0.06,0.07 is here! [!]
    By encode in forum Data Compression
    Replies: 34
    Last Post: 31st May 2009, 17:39
  3. BCM v0.05 is here! [!]
    By encode in forum Data Compression
    Replies: 19
    Last Post: 8th March 2009, 22:12
  4. BCM v0.04 is here! [!]
    By encode in forum Data Compression
    Replies: 64
    Last Post: 5th March 2009, 17:07
  5. BCM v0.03 is here! [!]
    By encode in forum Data Compression
    Replies: 25
    Last Post: 14th February 2009, 15:42

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •