Page 2 of 3 FirstFirst 123 LastLast
Results 31 to 60 of 65

Thread: BCM v0.04 is here! [!]

  1. #31
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Just to backup my words:
    Look at metacompressor and text tests big enough to get quite reliable decompression speed measurements. There is always (FreeArc) LZP+PPMD with some parameters that's stronger and decompresses faster than BCM.
    Sometimes (File3:log) the difference is really huge.

  2. #32
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    I have an idea about adding compression modes:
    9 - 320 MB block
    8 - 160 MB block
    7 - 80 MB block
    6 - 40 MB block
    5 - 20 MB block
    4 - 10 MB block
    3 - 5 MB block
    2 - 2.5 MB block
    1 - 1.25 MB block



    Along with an improved CM, this might be really cool!

  3. #33
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Are we really need such a big block size (>64 MiB) for any practical purposes? Or do you target some benchmarks for beating the other competitor with more memory? For me, 64 MiB at max is enough. So, memory requirements will be 5n=5x64=320+your context model. It's enough big already.
    BIT Archiver homepage: www.osmanturan.com

  4. #34
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    Quote Originally Posted by osmanturan View Post
    Are we really need such a big block size (>64 MiB) for any practical purposes? Or do you target some benchmarks for beating the other competitor with more memory? For me, 64 MiB at max is enough. So, memory requirements will be 5n=5x64=320+your context model. It's enough big already.
    Or even better I'll add an option of a block size in MB. Yep, why Blizzard can be tested with 333 MB block and BCM not?
    Anyway, you may set a block size to any number you want.

  5. #35
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    BTW, even new context model is small enough - ~700 KB

  6. #36
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    BCM v0.05, option 9:

    ENWIK9 -> 172,180,796 bytes


  7. #37
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    412
    Thanks
    38
    Thanked 64 Times in 38 Posts
    encode wrote:
    ---
    Anyway, you may set a block size to any number you want.
    ---

    sounds wonderful

    we are ready to test

  8. #38
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    215
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by encode View Post
    BCM v0.05, option 9:

    ENWIK9 -> 172,180,796 bytes

    Wow One of the best BWT to date?.. nz -cO is your only competitor I don't take BBB cause it's waaay too slow for r/l

    Good job!
    Last edited by nanoflooder; 18th February 2009 at 15:42.

  9. #39
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    Quote Originally Posted by nanoflooder View Post
    Wow One of the best BWT to date?.. nz -cO is your only competitor I don't take BBB cause it's waaay too slow for r/l

    Good job!
    BCM with 1GB block will beat BBB! But we need a 64-bit system with more than 5 GB of RAM installed. And NZ may beat BCM due to its text preprocessing only, AFAIK.
    Here, option 9 means 320 MB block. Also I'll test it, say, with 350 MB one.

  10. #40
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    BTW, CCMX and even CMM4 are beaten as well...

  11. #41
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,564
    Thanks
    773
    Thanked 687 Times in 372 Posts
    why don't compile 64-bit version? i sure there are people here with 16gb of ram

  12. #42
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    why don't compile 64-bit version? i sure there are people here with 16gb of ram
    No problem! But at the moment I'm busy optimizing new CM - it's not really a trivial task and takes a huge amount of time...
    If someone have at least 6 GB of RAM installed, please let me know...

  13. #43
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    412
    Thanks
    38
    Thanked 64 Times in 38 Posts
    i have a windows 2003 server 32bit
    and a win xp 32 bit with 4 GByte RAM
    so i can give full 2 GByte to the application
    - want to test

  14. #44
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    Quote Originally Posted by joerg View Post
    i have a windows 2003 server 32bit
    and a win xp 32 bit with 4 GByte RAM
    so i can give full 2 GByte to the application
    - want to test
    New BCM will have no block size limit.

    Code:
    usage: bcm e[nn]|d in out
    or
    usage: bcm e[n]|d in out
    or
    usage: bcm e[#]|d in out
    Vote for the default block size! I think it should be about 5 MB.

  15. #45
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    My homemate has a Q6600 with 6 GiB RAM But, I won't be in my flat until 2 March. Maybe you can finish the other improvements until that date.
    BIT Archiver homepage: www.osmanturan.com

  16. #46
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by encode View Post
    New BCM will have no block size limit.

    Code:
    usage: bcm e[nn]|d in out
    Vote for the default block size! I think it should be about 5 MB.
    120 MB?

  17. #47
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    Quote Originally Posted by m^2 View Post
    120 MB?
    120*5=600 MB - too heavy. 8/16 MB block can be better. An older BWT compressors have 4/5 MB block, taking into account modern hardware we may use a larger default block.

  18. #48
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    mcomp has a 64MB block by default... good idea!

  19. #49
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by encode View Post
    120*5=600 MB - too heavy. 8/16 MB block can be better. An older BWT compressors have 4/5 MB block, taking into account modern hardware we may use a larger default block.
    That's exactly how I calculated.
    Come on, it's experimental compressor, compatibility with 98% of computers is enough.

  20. #50
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    Quote Originally Posted by m^2 View Post
    That's exactly how I calculated.
    Come on, it's experimental compressor, compatibility with 98% of computers is enough.
    Yep, maybe you're right!

    Anyway, the user may set any block size in MB, e.g.

    bcm e400 enwik9 enwik9.bcm

  21. #51
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    Or maybe add a mem option - i.e. mem usage.

    say

    bcm e1700 enwik9 enwik9.bcm

    block size = mem/5

    NanoZip like approach. This can be more useful. What do you think?

  22. #52
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    412
    Thanks
    38
    Thanked 64 Times in 38 Posts
    i think

    bcm e nnnn source target.bcm

    will be nice

    ask: which amount of memory is the minimum to work?

    5 MByte - 1 MByte blocksize ??

    for test purposes it would be wonderful
    if the program can at start write to the console
    the amount of available memory
    and the amount of used memory

    as default i propose

    nnnn = 0960 - 960 MByte - 192 MByte blocksize

    if there is not enough memory available, then
    maybe do a automatic fallback to

    nnnn = 0320 - 320 MByte - 64 MByte blocksize

    if this amount of memory is too not available, then
    maybe do a automatic fallback to

    nnnn = 0040 - 40 MByte - 8 MByte blocksize
    ...
    nnnn = 0005 - 5 MByte - 1 MByte blocksize

    best regards

  23. #53
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    Quote Originally Posted by joerg View Post
    ask: which amount of memory is the minimum to work?
    Actually, I wrote a special BCM lite - which uses 193 KB of memory, compressing better than ZIP.

    Quote Originally Posted by joerg View Post
    5 MByte - 1 MByte blocksize ??
    With mem option - yes. But I still prefer the block size in MB...

  24. #54
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    BCM with various block sizes on ENWIK9:

    64m -> 185,309,765 bytes
    120m -> 180,155,865 bytes
    300m -> 173,440,311 bytes
    310m -> 172,934,216 bytes
    340m -> 172,219,709 bytes
    333m -> 172,214,743 bytes
    350m -> 172,209,734 bytes
    360m -> 172,185,320 bytes
    320m -> 172,180,796 bytes
    370m -> 172,125,800 bytes
    375m -> 172,091,083 bytes


  25. #55
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    Tested, 13,616,873 0.03 failed decompression at one file and hasn't been listed.
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  26. #56
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    Quote Originally Posted by Black_Fox View Post
    Tested, 13,616,873
    Thanks a lot!

    Quote Originally Posted by Black_Fox View Post
    0.03 failed decompression at one file and hasn't been listed.
    It's thread about v0.04!

  27. #57
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    222
    Thanked 146 Times in 83 Posts

    Hi Ilia!

    MOC Test passed for last version ! Very good results !

  28. #58
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,010
    Thanks
    399
    Thanked 398 Times in 152 Posts
    Actually, new BCM's compression engine is done a long time ago. Just currently I have too little spare time...
    Anyway, finally, BCM will have a proper GZIP-like command-line interface. Strangely, if I worked so hard on it's compression engine, running optimizer for a few weeks and so, and at the same time kept such dummy command-line interface.
    Code:
    bcm [options] input [output]
    As expected, -d means decode/decompress. -f means force overwrite of output file. The -b option sets the block size. Not decided yet about, should user specify the block size in KB or MB or in both [k,m]. Also not decided yet about the E8/E9 filter - should I remove it or add a new filter that will work with a large blocks...


  29. #59
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    412
    Thanks
    38
    Thanked 64 Times in 38 Posts
    if you think the blocksize as parameter is better - why not
    then
    i propose to specify blocksize in KiloBytes

    for example -b393216
    means blocksize = 393216 KB = 384 MB .. using 1920 MByte RAM

    but wishes

    1. a option -V = verbose
    would be wonderful for test purposes
    "program writes at start to the console
    the amount of available memory
    and the amount of used memory"

    2. if blocksize option is not selected
    then i propose
    default blocksize = 192 MByte .. using 960 MByte RAM

    with an automatic fallback to a lower blocksize
    if there is not enough memory available

    blocksize = 65536 KB .. using 320 MByte RAM

    if this amount of memory is too not available,
    then maybe do a automatic fallback to

    blocksize = 8192 KB .. using 40 MByte RAM
    ..
    blocksize = 1024 KB .. using 5 MByte RAM

    3. if you have the time it would be very nice to have a

    option -r
    (recursivly compress a whole directory with subdirecty-tree)

    bcm -r input output

    where input is a directory like D:\PICTURES\

    4. have a option -nofilter
    to switch off the E8/E9 filter

    best regards

  30. #60
    Member chornobyl's Avatar
    Join Date
    May 2008
    Location
    ua/kiev
    Posts
    153
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I would prefer block size in MEGAbytes (not much people will use blocks under 1 MB imho).

Page 2 of 3 FirstFirst 123 LastLast

Similar Threads

  1. BCM v0.10 is here!
    By encode in forum Data Compression
    Replies: 45
    Last Post: 20th June 2010, 21:39
  2. BCM's future
    By encode in forum Data Compression
    Replies: 17
    Last Post: 9th August 2009, 01:00
  3. BCM v0.06,0.07 is here! [!]
    By encode in forum Data Compression
    Replies: 34
    Last Post: 31st May 2009, 16:39
  4. BCM v0.05 is here! [!]
    By encode in forum Data Compression
    Replies: 19
    Last Post: 8th March 2009, 21:12
  5. BCM v0.03 is here! [!]
    By encode in forum Data Compression
    Replies: 25
    Last Post: 14th February 2009, 14:42

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •