BALZ - An Open-Source ROLZ-based compressor

[encode]

Well, this is a special thread about BALZ.

The compression part at almost is done - I simplified some things to get better speed, however compression is still good. Very soon I'll release a prototype of future open-source project.

A have to ask a few questions to the audience:

1. License type. I'm thinking about GPL/LGPL/Public Domain. I consider that people don't care much about licenses - even if a project was released under GPL/LGPL people rip out the code without a credit and write something like this thing is based on XXX...

2. Command-line user interface - a la GZIP/BZIP or standard e[x]/d?

3. If you're Linux programmer, can you provide synonym of _filelengthi64() for Linux/BSD/... systems?

All in all, new BALZ is the QUAD replacement - portable, open source, with higher compression and much faster decompression...

[LovePimple]

1. License type. I'm thinking about GPL/LGPL/Public Domain. I consider that people don't care much about licenses - even if a project was released under GPL/LGPL people rip out the code without a credit and write something like this thing is based on XXX...

Anything that is released to the public as open source, PD etc is always at risk of abuse. We need look no further than Infima Archiver for a prime example of this.

Most people seem to prefer LGPL licensing.

2. Command-line user interface - a la GZIP/BZIP or standard e[x]/d?

I much prefer the standard (balz e[x]|d in out) command-line interface.

[Bulat Ziganshin]

static inline off_t myfilelength (int h)
{
off_t saved = lseek (h, 0, SEEK_CUR);
off_t size = lseek (h, 0, SEEK_END);
lseek (h, saved, SEEK_SET);
return size;
}

ream man off_t - afaik there is some way to make it 64-bit

[encode]

OK, I think that new and final BALZ will be something like this:
balz114.zip

Tell me what do you think!

[osmanturan]

This one is seems better than BALZ 1.13. Good work. Here is the some test results:

a10.jpg -> 836,321 bytes
AcroRd32.exe -> 1,451,427 bytes
english.dic -> 766,554 bytes
FlashMX.pdf -> 3,756,983 bytes
FP.log -> 577,291 bytes
MSO97.dll -> 1,877,014 bytes
ohs.doc -> 824,656 bytes
rafale.bmp -> 1,018,646 bytes
vcfiu.hlp -> 667,514 bytes
world95.txt -> 588,480 bytes

Total: 12,364,886 bytes

Compressing time: 30.285 Seconds (measured with AcuTimer 1.2)

Test platform: Core2 Duo 2.2 GHz, 2 GB RAM, Vista Business x64 SP1

I only found that rafale.bmp seems a worst-case for new BALZ. But, I didn't take it serious. Because, it's a MM file and can be handled by another way I guess there is no any filters in this release - even your well used exe filter. Am I wrong?

[encode]

I guess there is no any filters in this release - even your well used exe filter. Am I wrong?

Well, it has an EXE filter inside...

[osmanturan]

I forget to answer your questions. Here is my opinions:

1. License type. I'm thinking about GPL/LGPL/Public Domain. I consider that people don't care much about licenses - even if a project was released under GPL/LGPL people rip out the code without a credit and write something like this thing is based on XXX...

Personelly, I don't mind about licensing. Because, when I get an open source project, I only try to learn a new algorithm for new ideas - BIT 0.1 is exception, it's kind of a toy which based on modified QUAD (only parsing schema is same in BIT 0.1). I have learned ROLZ among compression basis by playing your code. Thanks again. Now, I have new ideas to implement a ROLZ codec in my compressor (=BIT) besides LWCX codec.

2. Command-line user interface - a la GZIP/BZIP or standard e[x]/d?

"e[x]/d" is better for me.

3. If you're Linux programmer, can you provide synonym of _filelengthi64() for Linux/BSD/... systems?

By looking your PIM project I can guess your are a delphi programmer too. In System.pas there must be a 64-bit version of it for both Win32 and Linux platforms. In delphi, everything is highly compatible with any situations. So, it's better to cheat file operations from System.pas

[osmanturan]

Well, it has an EXE filter inside...

Really? I was disappointed in here. Maybe there is something wrong about exe filter. In my TODO list, there is an exe filter too. If I'm able to do a better filter, I want to share with you and the others

[LovePimple]

Thanks Ilia! I would be happy with this as the final release.

[toffer]

Are you going to start a new CM compressor afterwards?

[osmanturan]

Are you going to start a new CM compressor afterwards?

This is also my wishes

[Intrinsic]

I simplified some things to get better speed, however compression is still good.

I'm glad to hear this! As i wasn't so keen on the direction BALZ took after 1.07(it just got less desirable to me, yes a bit better compression, but the speed took a huge hit)

[encode]

Are you going to start a new CM compressor afterwards?

Time will show... CM is quite complex - it's not so easy. Anyway, I already do some experiments with CM (FCM, BCM, and my non-public compressors)...

I just think that this BALZ is quite interesting - fast and efficient - worth to release it in favor to replace outdated QUAD...

[toffer]

It's still a good place to gather new knowledge. I'd be happy to see more people in this area. Somehow almost everything is too PAQish

Luckily i've got an outline for a new architecture, which supports context merging and benefits from "better compressible data", both in speed and compression. After my exams i'll start to implement it.

Sorry, this might not be the right thread, but i'd really like to see more people in this area.

[encode]

Found a way to eliminate _filelengthi64():

Write pad of header size

Write compressed data counting size

rewind() and write out the header

Just tested BALZ under Fedora Linux - installed it right on Flash Drive. It's cool - regular Linux on Flash Drive - it works just as HDD - I can freely boot via this pen-sized thing!

BALZ works correctly under Linux and is fully compatible with Windows version.

Now I need to fully comment the source code. I hate sources with no comments - LZMA for example - no one can understand what's going on inside.

[Black_Fox]

Tested. Very nice compression speed increase, thanks

[encode]

Tested. Very nice compression speed increase, thanks

Thanks for testing! Like I see, it's fastest from all versions!

[encode]

Also I think finally I will add a CRC32 checking...

[joerg]

sourcefile db.dmp 648.331.264 bytes

7zip 35.151.362 bytes 19 min

balz113 e 33.390.086 bytes 5 min
balz113 ex 30.707.411 bytes 38 min

balz114 e 34.119.363 bytes 2 min
balz114 ex 32.248.798 bytes 9,5 min

it seems that you go
a big step in direction faster compression ?!

it will be interesting to see the new fast balz e -option
in other tests like mfc

thank you!

wonderful !

[encode]

The main benefit of the new BALZ is its decompression speed!

[encode]

Started writing comments and understood that its equal to "War & Peace" with French comments or huge book about Data Compression. Too many stuff and tricks inside which cannot be explained so simple for regular developer...
Will write comments next time...

[encode]

The coding part is done, at least for now. Waiting for SourceForge.net project approval.

[Christian]

Great that you release BALZ as OSS! Btw. I like the new version much better than 1.13 because it 'feels' much more efficient.

Will write comments next time...

[encode]

BTW, the source code will be in Public Domain!

[Fallon]

Well, one would have to be a moron not to like some of the releases by Christian, Toffer and Francesco on this forum, but I must admit that I care more about an open source release like this cross platform Balz (and projects like Freearc and Peazip), than about most of the others.

The main benefit of the new BALZ is its decompression speed!

Black_Fox's testpage reveals it nicely!

If I don't have a chance to be online later, thank you very much already and congrats.

[Shelwien]

http://ctxmodel.net/rem.pl?-24
http://ctxmodel.net/files/MIX/mix_v0.htm
Accidentally, there's a balz 1.14 benchmark too.
It seems that this version has compression speed and
quality similar to rar -m5 -mdg -mct- , with 2x slower decoding speed.

Also it seems that my finn.lst is a really interesting sample -
paq9a compresses it 3x+ better than balz.

[encode]

http://ctxmodel.net/rem.pl?-24
http://ctxmodel.net/files/MIX/mix_v0.htm
Accidentally, there's a balz 1.14 benchmark too.
It seems that this version has compression speed and
quality similar to rar -m5 -mdg -mct- , with 2x slower decoding speed.

Also it seems that my finn.lst is a really interesting sample -
paq9a compresses it 3x+ better than balz.

We should carefully compare BALZ to other compressors, like RAR, since it uses completely different algorithm with its own strong and weak parts.

[encode]

Hm, this time SF.net staff much like turtles... BALZ is still not approved... Waiting...

Well, and now, please check the README.TXT!

[encode]

Hooray! I did it!

Welcome people:
http://balz.sourceforge.net/

[LovePimple]

Thanks Ilia!

Mirror: Download