Page 1 of 3 123 LastLast
Results 1 to 30 of 61

Thread: BALZ - An Open-Source ROLZ-based compressor

  1. #1
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts

    Cool BALZ - An Open-Source ROLZ-based compressor

    Well, this is a special thread about BALZ.

    The compression part at almost is done - I simplified some things to get better speed, however compression is still good. Very soon I'll release a prototype of future open-source project.

    A have to ask a few questions to the audience:

    1. License type. I'm thinking about GPL/LGPL/Public Domain. I consider that people don't care much about licenses - even if a project was released under GPL/LGPL people rip out the code without a credit and write something like this thing is based on XXX...

    2. Command-line user interface - a la GZIP/BZIP or standard e[x]/d?

    3. If you're Linux programmer, can you provide synonym of _filelengthi64() for Linux/BSD/... systems?

    All in all, new BALZ is the QUAD replacement - portable, open source, with higher compression and much faster decompression...

  2. #2
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by encode
    1. License type. I'm thinking about GPL/LGPL/Public Domain. I consider that people don't care much about licenses - even if a project was released under GPL/LGPL people rip out the code without a credit and write something like this thing is based on XXX...
    Anything that is released to the public as open source, PD etc is always at risk of abuse. We need look no further than Infima Archiver for a prime example of this.

    Most people seem to prefer LGPL licensing.


    Quote Originally Posted by encode
    2. Command-line user interface - a la GZIP/BZIP or standard e[x]/d?
    I much prefer the standard (balz e[x]|d in out) command-line interface.

  3. #3
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    static inline off_t myfilelength (int h)
    {
    off_t saved = lseek (h, 0, SEEK_CUR);
    off_t size = lseek (h, 0, SEEK_END);
    lseek (h, saved, SEEK_SET);
    return size;
    }

    ream man off_t - afaik there is some way to make it 64-bit

  4. #4
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts

    Exclamation

    OK, I think that new and final BALZ will be something like this:
    balz114.zip

    Tell me what do you think!

  5. #5
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    This one is seems better than BALZ 1.13. Good work. Here is the some test results:

    Code:
    a10.jpg -> 836,321 bytes
    AcroRd32.exe -> 1,451,427 bytes
    english.dic -> 766,554 bytes
    FlashMX.pdf -> 3,756,983 bytes
    FP.log -> 577,291 bytes
    MSO97.dll -> 1,877,014 bytes
    ohs.doc -> 824,656 bytes
    rafale.bmp -> 1,018,646 bytes
    vcfiu.hlp -> 667,514 bytes
    world95.txt -> 588,480 bytes
    
    Total: 12,364,886 bytes
    
    Compressing time: 30.285 Seconds (measured with AcuTimer 1.2)
    Test platform: Core2 Duo 2.2 GHz, 2 GB RAM, Vista Business x64 SP1

    I only found that rafale.bmp seems a worst-case for new BALZ. But, I didn't take it serious. Because, it's a MM file and can be handled by another way I guess there is no any filters in this release - even your well used exe filter. Am I wrong?

  6. #6
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts
    Quote Originally Posted by osmanturan View Post
    I guess there is no any filters in this release - even your well used exe filter. Am I wrong?
    Well, it has an EXE filter inside...

  7. #7
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I forget to answer your questions. Here is my opinions:

    Quote Originally Posted by encode
    1. License type. I'm thinking about GPL/LGPL/Public Domain. I consider that people don't care much about licenses - even if a project was released under GPL/LGPL people rip out the code without a credit and write something like this thing is based on XXX...
    Personelly, I don't mind about licensing. Because, when I get an open source project, I only try to learn a new algorithm for new ideas - BIT 0.1 is exception, it's kind of a toy which based on modified QUAD (only parsing schema is same in BIT 0.1). I have learned ROLZ among compression basis by playing your code. Thanks again. Now, I have new ideas to implement a ROLZ codec in my compressor (=BIT) besides LWCX codec.

    Quote Originally Posted by encode
    2. Command-line user interface - a la GZIP/BZIP or standard e[x]/d?
    "e[x]/d" is better for me.

    Quote Originally Posted by encode
    3. If you're Linux programmer, can you provide synonym of _filelengthi64() for Linux/BSD/... systems?
    By looking your PIM project I can guess your are a delphi programmer too. In System.pas there must be a 64-bit version of it for both Win32 and Linux platforms. In delphi, everything is highly compatible with any situations. So, it's better to cheat file operations from System.pas

  8. #8
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by encode View Post
    Well, it has an EXE filter inside...
    Really? I was disappointed in here. Maybe there is something wrong about exe filter. In my TODO list, there is an exe filter too. If I'm able to do a better filter, I want to share with you and the others

  9. #9
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Ilia! I would be happy with this as the final release.

  10. #10
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Are you going to start a new CM compressor afterwards?

  11. #11
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by toffer View Post
    Are you going to start a new CM compressor afterwards?
    This is also my wishes

  12. #12
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    Quote Originally Posted by encode View Post
    I simplified some things to get better speed, however compression is still good.
    I'm glad to hear this! As i wasn't so keen on the direction BALZ took after 1.07(it just got less desirable to me, yes a bit better compression, but the speed took a huge hit)

  13. #13
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts
    Quote Originally Posted by toffer View Post
    Are you going to start a new CM compressor afterwards?
    Time will show... CM is quite complex - it's not so easy. Anyway, I already do some experiments with CM (FCM, BCM, and my non-public compressors)...

    I just think that this BALZ is quite interesting - fast and efficient - worth to release it in favor to replace outdated QUAD...

  14. #14
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    It's still a good place to gather new knowledge. I'd be happy to see more people in this area. Somehow almost everything is too PAQish

    Luckily i've got an outline for a new architecture, which supports context merging and benefits from "better compressible data", both in speed and compression. After my exams i'll start to implement it.

    Sorry, this might not be the right thread, but i'd really like to see more people in this area.
    Last edited by toffer; 4th July 2008 at 00:31.

  15. #15
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts
    Found a way to eliminate _filelengthi64():

    Write pad of header size

    Write compressed data counting size

    rewind() and write out the header

    Just tested BALZ under Fedora Linux - installed it right on Flash Drive. It's cool - regular Linux on Flash Drive - it works just as HDD - I can freely boot via this pen-sized thing!

    BALZ works correctly under Linux and is fully compatible with Windows version.

    Now I need to fully comment the source code. I hate sources with no comments - LZMA for example - no one can understand what's going on inside.

  16. #16
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    Tested. Very nice compression speed increase, thanks
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  17. #17
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts
    Quote Originally Posted by Black_Fox View Post
    Tested. Very nice compression speed increase, thanks
    Thanks for testing! Like I see, it's fastest from all versions!

  18. #18
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts
    Also I think finally I will add a CRC32 checking...

  19. #19
    Member
    Join Date
    May 2008
    Location
    Germany
    Posts
    410
    Thanks
    37
    Thanked 60 Times in 37 Posts
    sourcefile db.dmp 648.331.264 bytes

    7zip 35.151.362 bytes 19 min

    balz113 e 33.390.086 bytes 5 min
    balz113 ex 30.707.411 bytes 38 min

    balz114 e 34.119.363 bytes 2 min
    balz114 ex 32.248.798 bytes 9,5 min

    it seems that you go
    a big step in direction faster compression ?!

    it will be interesting to see the new fast balz e -option
    in other tests like mfc

    thank you!

    wonderful !

  20. #20
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts
    The main benefit of the new BALZ is its decompression speed!

  21. #21
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts
    Started writing comments and understood that its equal to "War & Peace" with French comments or huge book about Data Compression. Too many stuff and tricks inside which cannot be explained so simple for regular developer...
    Will write comments next time...

  22. #22
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts
    The coding part is done, at least for now. Waiting for SourceForge.net project approval.

  23. #23
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 151 Times in 18 Posts
    Great that you release BALZ as OSS! Btw. I like the new version much better than 1.13 because it 'feels' much more efficient.

    Quote Originally Posted by encode View Post
    Will write comments next time...

  24. #24
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts
    BTW, the source code will be in Public Domain!

  25. #25
    Member Fallon's Avatar
    Join Date
    May 2008
    Location
    Europe - The Netherlands
    Posts
    155
    Thanks
    14
    Thanked 10 Times in 5 Posts

    Arrow

    Well, one would have to be a moron not to like some of the releases by Christian, Toffer and Francesco on this forum, but I must admit that I care more about an open source release like this cross platform Balz (and projects like Freearc and Peazip), than about most of the others.
    Quote Originally Posted by encode View Post
    The main benefit of the new BALZ is its decompression speed!
    Black_Fox's testpage reveals it nicely!

    If I don't have a chance to be online later, thank you very much already and congrats.

  26. #26
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,239
    Thanks
    192
    Thanked 968 Times in 501 Posts
    http://ctxmodel.net/rem.pl?-24
    http://ctxmodel.net/files/MIX/mix_v0.htm
    Accidentally, there's a balz 1.14 benchmark too.
    It seems that this version has compression speed and
    quality similar to rar -m5 -mdg -mct- , with 2x slower decoding speed.

    Also it seems that my finn.lst is a really interesting sample -
    paq9a compresses it 3x+ better than balz.

  27. #27
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts
    Quote Originally Posted by Shelwien View Post
    http://ctxmodel.net/rem.pl?-24
    http://ctxmodel.net/files/MIX/mix_v0.htm
    Accidentally, there's a balz 1.14 benchmark too.
    It seems that this version has compression speed and
    quality similar to rar -m5 -mdg -mct- , with 2x slower decoding speed.

    Also it seems that my finn.lst is a really interesting sample -
    paq9a compresses it 3x+ better than balz.
    We should carefully compare BALZ to other compressors, like RAR, since it uses completely different algorithm with its own strong and weak parts.

  28. #28
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts
    Hm, this time SF.net staff much like turtles... BALZ is still not approved... Waiting...

    Well, and now, please check the README.TXT!

  29. #29
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,979
    Thanks
    376
    Thanked 347 Times in 137 Posts
    Hooray! I did it!

    Welcome people:
    http://balz.sourceforge.net/


  30. #30
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Ilia!

    Mirror: Download

Page 1 of 3 123 LastLast

Similar Threads

  1. Replies: 23
    Last Post: 24th March 2018, 17:57
  2. PeaZip - open source archiver
    By squxe in forum Data Compression
    Replies: 1
    Last Post: 3rd December 2009, 21:01
  3. fcm1 - open source order-1 cm encoder
    By encode in forum Data Compression
    Replies: 34
    Last Post: 4th June 2008, 23:16
  4. New fast open-source paq-based jpeg compressor
    By Bulat Ziganshin in forum Forum Archive
    Replies: 14
    Last Post: 13th September 2007, 13:57
  5. Replies: 0
    Last Post: 26th July 2007, 18:47

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •