LZA archiver

[Nania Francesco]

LZA is Windows/Dos archiver

Copyright (C) LZA Nania Francesco ( Yann Collet only 0.60-63 version).


LZA is a data archiver that uses a new efficient LZ77 algorithm with data analysis, filters for certain file types (WAV,BMP,PPM,PGM etc) or data (Delta coding) with:

- Simple and fast Bit Range Coder;
- Fast bit coder;
- adaptive *nArANS coder (is semi-adaptive range coder named nArANS created by Nania Francesco)
- fast and powerful match search method

From version 1.0 the code of LZA core cannot change as regards compatibility.
Given the great achievements the future release will be closed source.
last version is 0.82
download from:
http://heartofcomp.altervista.org/
Special thanks for Jarek, Yann, Matt, Stephan and Piotr.
*nArANS is an entropy coder from the ANS family, as described in Jarek Duda's paper "Asymmetric numeral systems" (http://arxiv.org/abs/1311.2540)

[Piotr Tarsa]

I suppose there should be a link to something, shouldn't be?

Edit:
OK, you've finally posted it

[m^2]

What exactly do you want to open source? As I see it, you've cut pieces in the middle of algorithms, making the code dump useless. If it's useless for everyone, so it's for potential collaborators. So unless I miss something you'll have hard time finding them.

I would like to work with somebody on a project that's close to yours. But not exactly. I don't think that world needs yet another LZ77, there's a rich selection of quality ones. What is missing is a fast lz with entropy coding that's no more than 3 times slower than lz4 on compression and still strongly asymmetric. Well, to be strict, I'm aware of 2 codecs like that. lzhuff which is closed source and therefore useless. And gipfeli, which is open source, but doesn't work. And now we have a great time to start such project. ANS gave us a chance to reach efficiency limits that were unachievable before. It's still very fresh, yet we understand it well enough to start using it in practice.

[Nania Francesco]

@m^2
What you say is partly right. There is currently no LZ compressor that reaches these levels of compression with a compression rate as great as LZA. According to fact and is more important not to consider that is just born project and already you scraps. Take a look at the results on WCC and you'll notice that as LZ77 + BYTE coder (not bit coder or arithmetic) with these benefits do not see. You can judge good seeing the results in other benchmarks. I gave him a chance to see the operation with partial code that replace with a single compressor source files fully compatible. At the end of the project with the version 1.0 project will be completely Open Source as a ZIP.

[Matt Mahoney]

I agree partial source code is not very useful. I think for an archive format to be useful and accepted then there should be a specification of the archive format or at least a complete open source decompressor.

Also, you might want to include a license. By default you own all copyright rights.

[Nania Francesco]

Have you actually reason is not useful to put a program in open source.
LZA is closed source until version 1.0

[Black_Fox]

From version 1.0 the code cannot change as regards compatibility.

LZA is closed source until version 1.0

That means only you shape the future of the archive format. Is that what you intend?

[Nania Francesco]

With this program LZA i was planning originally to share the main encoder code with everyone. The comments I received made me realize that I was wrong to put only part of the code. In fact I don't want to see all the code of the archiver. Release future releases closed-source but with version 1.0 I'll release the code and will become eventually Open Source (like ZIP). In the copyright which is able to explain actions only on the structure without affecting compatibility. I hope that the translator has helped me to explain.

[m^2]

@m^2
What you say is partly right. There is currently no LZ compressor that reaches these levels of compression with a compression rate as great as LZA. According to fact and is more important not to consider that is just born project and already you scraps. Take a look at the results on WCC and you'll notice that as LZ77 + BYTE coder (not bit coder or arithmetic) with these benefits do not see. You can judge good seeing the results in other benchmarks. I gave him a chance to see the operation with partial code that replace with a single compressor source files fully compatible. At the end of the project with the version 1.0 project will be completely Open Source as a ZIP.

I haven't scrapped it, I complain because I see something potentially good, but handled in a way that makes it useless. I'm very glad to read that you intend to open it fully.
As to wcc, it misses some important coders like encode's crush and LZSS and modes - like the stronger Tornado ones. But LZA does look fine.

[SvenBent]

LZA archiver v.0.01.  Warning is demo version.

copyright (C) 2014 By Nania Francesco Antonio

Archive is E:\Titan Quest + Immortal Throne\Game Discs\test.lza
 Compressed -2935265280 bytes to 5576550784 bytes

probably need a bigger variable to contain original size
also some output onscreen when working would be nice i was in doubt if it was running or frozen. (but i get that 0.01 is really basics only)

Size
RZM 5.14 GB (5,524,453,835 bytes)
LZA 5.19 GB (5,576,550,784 bytes)

decompression not tested yet.

[Nania Francesco]

with the next version I'll definitely to a hybrid bitcode + bytecode (literal). Current tests show an interesting improvement without losing appreciable decompression speed (1-5%)

[Nania Francesco]

I managed to improve a decent percentage in compression. Sure feels the lack of a coder for the scraps of the compression (usually referred to as literal). Decompression speed is 3-8% slower but is acceptable. We should accelerate the coder who is this only for decision, match and distance

u32 wbuf; // pending output in low bits
unsigned obj; // input bits, LSB first
unsigned n=0; // number of bits in bits (0..32)

u32 mask[29]={1,3,7,15,31,63,127,255,511,1023,2047,4095,8191,1 6383,32767,65535,131071,262143,524287,1048575,2097 151,4194303,8388607,16777215,33554431,67108863,134 217727,268435455,536870911};

void Wbits(unsigned x, int bit)
{ // write k (0..25) bits of x
x&=mask[bit-1];
wbuf|=x<<n;
n+=bit;
while (n>7) AUXBuffer[ascritto++]=wbuf, wbuf>>=8, n-=8;
}


int Rbits(int bit)
{ // write k (0..25) bits of x
while (n<bit)
{
wbuf+=AUXBuffer[ascritto++]<<n;
n+=8;
}
obj=wbuf&mask[bit-1];
wbuf>>=bit;
n-=bit;
return obj;
}

void encode(bool x)
{ // write k (0..25) bits of x
wbuf|=x<<n;
n++;
if (n>7) AUXBuffer[ascritto++]=wbuf, wbuf>>=8, n-=8;
}


int decode()
{ // write k (0..25) bits of x
(!n)?(wbuf+=AUXBuffer[ascritto++]<<n, n + = 8 ) : (0);
obj=wbuf&1;
wbuf>>=1;
n--;
return obj;
}

void flush()
{ // write last byte
if (n>0) AUXBuffer[ascritto++]=wbuf, wbuf=0, n=0;
}

@ Matt
If possible test LZA archiver 0.01 in LTCB and 10GB Benchmark!
Hi!

[tnagylev]

LZA compress literals, some 2 or 3 character long string stored in one byte.

These:
---------------------------------------------------------------------------------------------------------------
th he in er an on the re or ti at te en es al is ar nd st ed nt it of le ri to ng io ic me as se co and ra de ion ve ro ot li la om ing ha ne ce ea si ta ma ou hi ll am el ca tio ch ent us ns na tr qu ge di ur be ec ia ni ol ac il ct rs mp ter ati pe et id nc ie rt fo ly lo ut un uo no quo uot pr ate ho em so lt ss pa ry im mo po ts ad mi ag vi ee for ci ig os ir wa sh ai su wi ul iv con ow op ev her ver fi
---------------------------------------------------------------------------------------------------------------
This improve compress rate of english texts.

Enwik8.txt

100 000 000 uncompressed
34 644 188 original lza
36 794 530 my modified version (I remove smallstring replace)

[Matt Mahoney]

http://heartofcomp.altervista.org/ has no link to LZA. All of the other links are broken.

[Nania Francesco]

Ok ! added in this page !
I changed the front page of the site where I put all the links without alternative pages that I left to WCC (MOC)
To resolve this problem just press "update" in your broswer !

[Matt Mahoney]

Website is working now. Thanks.

Edit: updated LTCB. http://mattmahoney.net/dc/text.html#2755
Some observations:
- The l command crashes (lza l enwik8.lza) in Ubuntu/Wine 1.6. Works OK in Win32.
- Compression with -t2 is faster but worse than -t1 (like zcm).
- Decompression with -t2 takes twice as long as -t1.
- Options -h6 -b7 or -h7 -b6 (1.5 GB) causes "no memory" error in Wine (with 4 GB) but work OK in Win32. -h7 -b7 (2 GB) causes "no memory" error in both OS.

Edit: updated 10GB. http://mattmahoney.net/dc/10gb.html
Empty directories and file dates are not restored, but otherwise OK.

[Nania Francesco]

Good news:
- Added static Huffman bit coding only for match length with tables. Decompression is only 3-9 % slower than version 0.01!
- in some cases the efficiency during decompression is remarkable!

[SvenBent]

Do you consider implanting ECM prefiltering of cue+bin/ccd images ?
and what about PCM extractor into PCM compressor for cue+bin/ccd as well ?

[Nania Francesco]

I released version is already a data type equal to that of ZCM, recognizes generically rle images, audio, and put into an ISO or TAR or other type of file not compressed. You can certainly improve but it's very interesting.
LZA v.0.01 include 3 filter's for:
- WAVE/PCM
- BMP/PGM/PPM
- RLE DATA

[Mangix]

Do you consider implanting ECM prefiltering of cue+bin/ccd images ?
and what about PCM extractor into PCM compressor for cue+bin/ccd as well ?

This sounds pretty useless. Why would you even have cue+bin files instead of mds+iso?

[SvenBent]

You are the one claiming its pretty useless please enlighten me why.

but from my small gathering ecm would not work on a mds+iso as iso does not contain raw+96 sector in the first place but only the actual data layer,.
Without the raw+96 info you cannot stored the wrongly ECC layers that are needed to backup certain games..

If mds+iso does in fact store the raw+96 sectors then we are just debating what file format to put it in. never the less removing unneeded ECC data would still help compression


-- note --
actually i cant remember if cue/bin is able to store raw+96 or just raw and .CCD is needed to store raw+96. but i do believe mds+iso does not store raw at all.

[Mangix]

IIRC, ecm works by removing the ECC data from the files and recalculating/reapplying it upon decompressing.

ImgBurn (which is the only program that I use for optical media these days) will not allow you to back up a CD as cue/bin, only mds/iso. The only exception is audio CDs which have no ECC.

Personally, I've never encountered these games you speak of, but storing bin files instead of iso sounds like a waste of space.

[Nania Francesco]

the latest news regarding the new version doesn't differ much about the latest updates to the data structure of the coder.
The results are really interesting and in decompression better than option -cdp of Nanozip ( 5-10% in time of decompression and 5-10% in size )

[SvenBent]

IIRC, ecm works by removing the ECC data from the files and recalculating/reapplying it upon decompressing.

Exactly what i was saying. a RAW sector is a sector including the ECC part a RAW+96 is a sector including the ECC part and the 2 sub channels info as well
This EEC data is is only for data CD and is unrelated to the C1 and C2 error correction layer on a CD.
The c1 and C1 layer is also present on an audioCD

ImgBurn (which is the only program that I use for optical media these days) will not allow you to back up a CD as cue/bin, only mds/iso. The only exception is audio CDs which have no ECC.

I dont know why the program you decide to use constitutes a augment for calling another format for useless. very rarely does a program target at single user.

Personally, I've never encountered these games you speak of,

Again just because you never encountered them does not constitute a proof or even a valid argument they do not exist.
Fake ECC errors are among the earliest adaption of copy protection and one of the reasons CUE/bin adapted being able to read raw sectors.
The first many cd writers would read sector as data and thereby use the EEC data to correct a sector with was bad according to ECC and thereby "Fix" the sector
The software on execution would read the files places on such bad sectors and see if the files was intact or "Fixed"

Such a game among others is civilization 2 which is a game i treasure a lot.

its funny you refuse to believe the existing of such copy protection when even the the creatoer of ECM infomr that ECM does does in fact keep fake ECC errors in copy protected images.

What about copy protection? Don't they use bogus ECC data? Wouldn't ECM ruin that?

ECM only eliminates EDC/ECC data for sectors where it's verifiable possible to recreate that data. If the ECC is bogus, ECM will preserve that bogus data.


As extra info i a can inform as the next step many cd protections moved to mess around with the sub channels which was why it became a big fad with .CCD images which beside the raw sector a cue sheet also contained a file to store the sub channels. you have to make sure though you had read/write and software able to handle the raw+96 read and write mode ( data +ecc +96byts of sub channels)

next step in line was the tried to make hard to write EFM patterns. EFM is Eight to Fourteen .. something which is a bit pattern to bit pattern remapping to avoid bit patterns that could make the laser focus lose track. BY abusing the EFM patters they could make it hard for burner to reproduce the correct EFM patterns and thereby corrupt the data.
Copy protection today is more and more complex and newer images formats containing has been developed. like MDF/MDS
all of these have a purpose off describing more and more information about a cd images, and just because its not useable to your usage pattern does not make them objectively useless.

but storing bin files instead of iso sounds like a waste of space.

If its possible to store in Iso alone you are correct but for a lot of correct storing of CD images, ISO is useless (in the correct sense of the word)
By your sense of useless iso/mdf is useless as well since the mdf file is unneeded and you can store an CD image in just an iso file.


I hope in future you will be less inclined to let you lack of knowledge declare anything you don't know about as useless. it is a dangerous closeminded way to approach things.


Back on topic.
Im hoping to get some time to test LZA soon its looks like a promising project.

[Nania Francesco]

Released last version 0.10!

- Added static Huffman coding for match !
- Better compression (5- 20% )!
- 5-20 % slower than old version !
- LZ77 search core modified
- best option extended to -mx5

[Nania Francesco]

Result in Huge Files Compression Benchmark
File Vm.dll size=4.244.176.896 bytes

NANOZIP 0.07 -cd -m1.8g -> 1.173.141.528 bytes [benchmark results]

with Intel Core i7 920 2.67 ghz 4GB ram:
LZA v.0.10 -mx5 -b6 -h7 -> 1.179.091.707 bytes comp. 1154 sec. - decomp. 51 sec.
NANOZIP 0.09 -cdp -m1.8g -t1 ->1.181.462.334 comp. 288 sec. - decomp. 100 sec.

@ Matt
Please , test LZA v.0.10 in LTCB and 10GB benchmarks!

[Bloax]

3.4 GB/s?
Careful there, you might get a speeding ticket driving around like that!

[Matt Mahoney]

I think he means 1154 sec (. and , are reversed in U.S. and European numbers).

Anyway, test results show nice improvement:
http://mattmahoney.net/dc/text.html#2508
http://mattmahoney.net/dc/10gb.html

But "lza.exe l 10gb.lza" crashes (read page fault) after completing the listing under Wine 1.6 in 64 bit Ubuntu.

[Nania Francesco]

Thanks for all test Matt !
As Matt has already noticed the LZA version use the enhancements to the structure of data analysis I've implemented with ZCM. It should not seem strange that less effective Flashzip LZA, but just doesn't have a structure so evolved. The future version of Flashzip should be able to support in principle the results of ZCM.

[Stephan Busch]

When I use "lza a -r -s -v -mx5 -b7 -h7' the program tells me 'no memory'
Can this be fixed? Or is an 64-bit executable available?