balz v1.00 - new LZ77 encoder is here!

[encode]

Started writing a byte-wise LZ77 and adding an arithmetic compression, I came up to the new QUANTUM-like encoder:
+ Baseline LZ77 with 1 MB dictionary
+ Arithmetic encoding
+ Modified Storer&Szymanski parsing scheme
+ EXE transformer
Whatsoever, this new shit provides higher compression on binary files than LZPM, at the same time being faster at decompression.

Stop talking, check this out:
balz100.zip (46 KB)

But note, this peace can be VERY slow, especially on files like FP.LOG and on text files like ENWIK8/ENWIK9. So, test it on new machines like Core 2 Duo, you know what I'm saying...

Enjoy anyway!

[encode]

Just found one mistake in source, new version will be released ASAP, be patient!

[encode]

OK, here we are:
balz101.zip (46 KB)

[LovePimple]

Thanks Ilia!

But note, this peace can be VERY slow, especially on files like FP.LOG and on text files like ENWIK8/ENWIK9. So, test it on new machines like Core 2 Duo, you know what Im saying...

I may have to wait for members with powerful (expensive) machines to test this one though!

[encode]

I may have to wait for members with poweful (expensive) machines to test this one though!

If youre testing PAQ series on your machine - no worries!

[encode]

The thing with v1.00 was some "hidden" mistake - the coder just drops too many small matches. I'm afraid that all versions of LZPM have such bug too. Anyway, I believe that with an additional proper BALZ tuning we may get another compression gain - since all tunings was done on broken scheme. Will release another version after careful testing...

[vcore]

Hi , encode

Test of enwik8

My machine is Intel Core2 Duo Q6600 (Quad core) 2.4Ghz + DDR800 2GB RAM , Windows Vista 32bit

AcuTimer v1.2
Copyright (c) 2007 by LovePimple

balz v1.01 by encode
optimizing 16384k block...
optimizing 16384k block...
optimizing 16384k block...
optimizing 16384k block...
optimizing 16384k block...
optimizing 15736k block...
done

Elapsed Time: 00 00:35:56.813 (2156.813 Seconds)

final size---> 29,881,549 bytes

Okay, it used about 25~30% cpu load.
Well, the cost time is not accurate because I was using eMule during compression.

[encode]

Try to test it on binary files and TARs with lots of stuff.

[encode]

In addition, I will try to change the string searching method. Note that BALZ displays progress of string searching - actual parsing begins when compressor shows 100%. Like you see, parsing and coding takes no time, compared to the full string search...

[encode]

No extra testing results so far...

[Christian]

No extra testing results so far...

Here you go:
<div class=""jscript""><pre> 843.694 A10.balz
1.410.320 AcroRd32.balz
937.509 english.balz
3.733.912 FlashMX.balz
866.344 FP.balz
1.869.867 MSO97.balz
835.558 ohs.balz
1.035.397 rafale.balz
684.231 vcfiu.balz
612.546 world95.balz

-> 12.829.378 Bytes[/code]

I have not timed compression, but decompression speed is really pretty!

[Black_Fox1]

Well, running both BALZ and RZM on singlecore (and not very powerful) machine at the same time is sure fun

Due to not having test machine at hand only sizes without speeds follow.Thanks for release

14 228 209 - BALZ 1.01
13 318 039 - LZPM 0.15

[Shelwien]

> + Baseline LZ77 with 1 MB dictionary
> + Arithmetic encoding

Are you including EXE transformation choices into your optimization?
Or do you simply relocate 32bit suffixes of E8/E9 if there was a MZ header?
Well, I mean, that relocating or not is the same choice as with matches...

[encode]

EXE transformer works separately - the main encoder even not knows that actual transformation was done.
Transformer searches for 32-bit PE magic number (0x4550) in block and applies E8/E9 transformation after that point.

[LovePimple]

Quick test...

Test machine: Intel PIII (Coppermine) @750 MHz, 512 MB RAM, Windows 2000 Pro SP4

Test File: VALLEY.CMB (19,776,230 bytes)

Timed with AcuTimer v1.2

Compression
Compressed Size: 8,880,490 bytes
Elapsed Time: 00:18:49.767 (1129.767 Seconds)

Decompression
Elapsed Time: 00:00:06.514 (6.514 Seconds)

[encode]

Thanks!

[encode]

Looks like current version is OK in terms of carefully chosen parameters...

Made some experiments with version with lazy matching. Quite impressed by performance - compression speed is awesome, compression ratio is nice, but not so interesting as with SS parsing.

Anyway, looking forward for official benchmark results - MFC, Squeeze Chart, Black Fox's Benchmark, MOC, etc...

[encode]

The result on ENWIK9 is:
261,645,091 bytes

The compression took about four hours on my Core 2 Duo...

Well, I will release a version with Lazy Matching strategy. With some parsing tricks the compression ratio will be not that much worser, at the same time compression will be FAST!

[encode]

OK, the final "quick" release:
balz102.zip (46 KB)

This one is notable faster and has 512K window.

[Black_Fox1]

Thank you, has been tested as well
As window was halved, it's compressing twice as fast, also decompressing a little slower. Not much of performance change ratio-wise, of course except for pht/PSD, which is about 2,3MB bigger...

[LovePimple]

Thanks Ilia

[encode]

As window was halved, its compressing twice as fast,

On some files like world95.txt, fp.log, ENWIKs the compression is 4X-8X faster.

also decompressing a little slower.

In certain cases the decompression can be even faster...

[LovePimple]

Quick test...

A10.jpg > 843,681
AcroRd32.exe > 1,416,885
english.dic > 937,379
FlashMX.pdf > 3,740,800
FP.LOG > 896,743
MSO97.DLL > 1,893,199
ohs.doc > 838,427
rafale.bmp > 1,037,574
vcfiu.hlp > 685,817
world95.txt > 634,975

Total = 12,925,480 bytes

[encode]

Thanks!

[Nania Francesco]

Result MOC Test:
160.113.645
COMP TIME = 2.130,542 sec.
DEC. TIME=26,476 sec.
Hi Encode!

[encode]

You just retested the previous version! (Same compression results as with v1.01!)

[LovePimple]

Another quick test...

Test machine: Intel PIII (Coppermine) @750 MHz, 512 MB RAM, Windows 2000 Pro SP4

Test File: ENWIK8 (100,000,000 bytes)

Timed with AcuTimer v1.2

Compression
Compressed Size: 5,177,344 bytes
Elapsed Time: 00:36:40.436 (2200.436 Seconds)

Decompression
Elapsed Time: 00:00:20.729 (20.729 Seconds)

Decompressed file is longer (100,000,121 bytes) than the original.

[Nania Francesco]

Excuse Encode! Had copied Balz 1.02 in a different briefcase from that of the test! Ok I have tested ! Good improvements in the speed! be stable!
MOC Test: 161.766.723
Comp time=818,054 sec.
Dec. time=25,683 sec.
Hi !

[Nania Francesco]

Intel Core duo 2 E6600
Enwik8 : 30.634.726 b
Comp time= (397,994 Seconds) Acutimer
Dec time= (7,061 Seconds) Acutimer

[encode]

Decompressed file is longer (100,000,121 bytes) than the original.

Are you sure?? On my both PC and laptop decompression went OK!

Can anyone confirm the bug?