Page 1 of 7 123 ... LastLast
Results 1 to 30 of 201

Thread: LZA archiver

  1. #1
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts

    LZA archiver

    LZA is Windows/Dos archiver

    Copyright (C) LZA Nania Francesco ( Yann Collet only 0.60-63 version).


    LZA is a data archiver that uses a new efficient LZ77 algorithm with data analysis, filters for certain file types (WAV,BMP,PPM,PGM etc) or data (Delta coding) with:

    - Simple and fast Bit Range Coder;
    - Fast bit coder;
    - adaptive *nArANS coder (is semi-adaptive range coder named nArANS created by Nania Francesco)
    - fast and powerful match search method

    From version 1.0 the code of LZA core cannot change as regards compatibility.
    Given the great achievements the future release will be closed source.
    last version is 0.82
    download from:
    http://heartofcomp.altervista.org/
    Special thanks for Jarek, Yann, Matt, Stephan and Piotr.
    *nArANS is an entropy coder from the ANS family, as described in Jarek Duda's paper "Asymmetric numeral systems" (http://arxiv.org/abs/1311.2540)
    Last edited by Nania Francesco; 12th March 2015 at 00:26.

  2. #2
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,474
    Thanks
    26
    Thanked 121 Times in 95 Posts
    I suppose there should be a link to something, shouldn't be?

    Edit:
    OK, you've finally posted it
    Last edited by Piotr Tarsa; 29th May 2014 at 18:27.

  3. #3
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    What exactly do you want to open source? As I see it, you've cut pieces in the middle of algorithms, making the code dump useless. If it's useless for everyone, so it's for potential collaborators. So unless I miss something you'll have hard time finding them.

    I would like to work with somebody on a project that's close to yours. But not exactly. I don't think that world needs yet another LZ77, there's a rich selection of quality ones. What is missing is a fast lz with entropy coding that's no more than 3 times slower than lz4 on compression and still strongly asymmetric. Well, to be strict, I'm aware of 2 codecs like that. lzhuff which is closed source and therefore useless. And gipfeli, which is open source, but doesn't work. And now we have a great time to start such project. ANS gave us a chance to reach efficiency limits that were unachievable before. It's still very fresh, yet we understand it well enough to start using it in practice.

  4. #4
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    @m^2
    What you say is partly right. There is currently no LZ compressor that reaches these levels of compression with a compression rate as great as LZA. According to fact and is more important not to consider that is just born project and already you scraps. Take a look at the results on WCC and you'll notice that as LZ77 + BYTE coder (not bit coder or arithmetic) with these benefits do not see. You can judge good seeing the results in other benchmarks. I gave him a chance to see the operation with partial code that replace with a single compressor source files fully compatible. At the end of the project with the version 1.0 project will be completely Open Source as a ZIP.
    Last edited by Nania Francesco; 29th May 2014 at 22:20.

  5. #5
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    I agree partial source code is not very useful. I think for an archive format to be useful and accepted then there should be a specification of the archive format or at least a complete open source decompressor.

    Also, you might want to include a license. By default you own all copyright rights.

  6. #6
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Have you actually reason is not useful to put a program in open source.
    LZA is closed source until version 1.0
    Last edited by Nania Francesco; 29th May 2014 at 23:01.

  7. #7
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    Quote Originally Posted by Nania Francesco View Post
    From version 1.0 the code cannot change as regards compatibility.
    Quote Originally Posted by Nania Francesco View Post
    LZA is closed source until version 1.0
    That means only you shape the future of the archive format. Is that what you intend?
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  8. #8
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    With this program LZA i was planning originally to share the main encoder code with everyone. The comments I received made me realize that I was wrong to put only part of the code. In fact I don't want to see all the code of the archiver. Release future releases closed-source but with version 1.0 I'll release the code and will become eventually Open Source (like ZIP). In the copyright which is able to explain actions only on the structure without affecting compatibility. I hope that the translator has helped me to explain.

  9. #9
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Nania Francesco View Post
    @m^2
    What you say is partly right. There is currently no LZ compressor that reaches these levels of compression with a compression rate as great as LZA. According to fact and is more important not to consider that is just born project and already you scraps. Take a look at the results on WCC and you'll notice that as LZ77 + BYTE coder (not bit coder or arithmetic) with these benefits do not see. You can judge good seeing the results in other benchmarks. I gave him a chance to see the operation with partial code that replace with a single compressor source files fully compatible. At the end of the project with the version 1.0 project will be completely Open Source as a ZIP.
    I haven't scrapped it, I complain because I see something potentially good, but handled in a way that makes it useless. I'm very glad to read that you intend to open it fully.
    As to wcc, it misses some important coders like encode's crush and LZSS and modes - like the stronger Tornado ones. But LZA does look fine.

  10. #10
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    864
    Thanks
    46
    Thanked 104 Times in 82 Posts
    Code:
    LZA archiver v.0.01.  Warning is demo version.
    
    copyright (C) 2014 By Nania Francesco Antonio
    
    Archive is E:\Titan Quest + Immortal Throne\Game Discs\test.lza
     Compressed -2935265280 bytes to 5576550784 bytes
    probably need a bigger variable to contain original size
    also some output onscreen when working would be nice i was in doubt if it was running or frozen. (but i get that 0.01 is really basics only)
    Code:
    Size
    RZM 5.14 GB (5,524,453,835 bytes)
    LZA 5.19 GB (5,576,550,784 bytes)
    decompression not tested yet.
    Last edited by SvenBent; 31st May 2014 at 04:04.

  11. The Following User Says Thank You to SvenBent For This Useful Post:

    Nania Francesco (31st May 2014)

  12. #11
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    with the next version I'll definitely to a hybrid bitcode + bytecode (literal). Current tests show an interesting improvement without losing appreciable decompression speed (1-5%)
    Last edited by Nania Francesco; 1st June 2014 at 00:30.

  13. #12
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    I managed to improve a decent percentage in compression. Sure feels the lack of a coder for the scraps of the compression (usually referred to as literal). Decompression speed is 3-8% slower but is acceptable. We should accelerate the coder who is this only for decision, match and distance
    u32 wbuf; // pending output in low bits
    unsigned obj; // input bits, LSB first
    unsigned n=0; // number of bits in bits (0..32)

    u32 mask[29]={1,3,7,15,31,63,127,255,511,1023,2047,4095,8191,1 6383,32767,65535,131071,262143,524287,1048575,2097 151,4194303,8388607,16777215,33554431,67108863,134 217727,268435455,536870911};

    void Wbits(unsigned x, int bit)
    { // write k (0..25) bits of x
    x&=mask[bit-1];
    wbuf|=x<<n;
    n+=bit;
    while (n>7) AUXBuffer[ascritto++]=wbuf, wbuf>>=8, n-=8;
    }


    int Rbits(int bit)
    { // write k (0..25) bits of x
    while (n<bit)
    {
    wbuf+=AUXBuffer[ascritto++]<<n;
    n+=8;
    }
    obj=wbuf&mask[bit-1];
    wbuf>>=bit;
    n-=bit;
    return obj;
    }

    void encode(bool x)
    { // write k (0..25) bits of x
    wbuf|=x<<n;
    n++;
    if (n>7) AUXBuffer[ascritto++]=wbuf, wbuf>>=8, n-=8;
    }


    int decode()
    { // write k (0..25) bits of x
    (!n)?(wbuf+=AUXBuffer[ascritto++]<<n, n + = 8 ) : (0);
    obj=wbuf&1;
    wbuf>>=1;
    n--;
    return obj;
    }

    void flush()
    { // write last byte
    if (n>0) AUXBuffer[ascritto++]=wbuf, wbuf=0, n=0;
    }
    @ Matt
    If possible test LZA archiver 0.01 in LTCB and 10GB Benchmark!
    Hi!
    Last edited by Nania Francesco; 1st June 2014 at 14:36.

  14. #13
    Member
    Join Date
    Jun 2014
    Location
    Hungary
    Posts
    1
    Thanks
    1
    Thanked 3 Times in 1 Post
    LZA compress literals, some 2 or 3 character long string stored in one byte.

    These:
    ---------------------------------------------------------------------------------------------------------------
    th he in er an on the re or ti at te en es al is ar nd st ed nt it of le ri to ng io ic me as se co and ra de ion ve ro ot li la om ing ha ne ce ea si ta ma ou hi ll am el ca tio ch ent us ns na tr qu ge di ur be ec ia ni ol ac il ct rs mp ter ati pe et id nc ie rt fo ly lo ut un uo no quo uot pr ate ho em so lt ss pa ry im mo po ts ad mi ag vi ee for ci ig os ir wa sh ai su wi ul iv con ow op ev her ver fi
    ---------------------------------------------------------------------------------------------------------------
    This improve compress rate of english texts.

    Enwik8.txt

    100 000 000 uncompressed
    34 644 188 original lza
    36 794 530 my modified version (I remove smallstring replace)
    Last edited by tnagylev; 1st June 2014 at 23:46.

  15. The Following 3 Users Say Thank You to tnagylev For This Useful Post:

    Bulat Ziganshin (2nd June 2014),Nania Francesco (1st June 2014),Paul W. (5th June 2014)

  16. #14
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    http://heartofcomp.altervista.org/ has no link to LZA. All of the other links are broken.

  17. #15
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Ok ! added in this page !
    I changed the front page of the site where I put all the links without alternative pages that I left to WCC (MOC)
    To resolve this problem just press "update" in your broswer !

  18. #16
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    Website is working now. Thanks.

    Edit: updated LTCB. http://mattmahoney.net/dc/text.html#2755
    Some observations:
    - The l command crashes (lza l enwik8.lza) in Ubuntu/Wine 1.6. Works OK in Win32.
    - Compression with -t2 is faster but worse than -t1 (like zcm).
    - Decompression with -t2 takes twice as long as -t1.
    - Options -h6 -b7 or -h7 -b6 (1.5 GB) causes "no memory" error in Wine (with 4 GB) but work OK in Win32. -h7 -b7 (2 GB) causes "no memory" error in both OS.

    Edit: updated 10GB. http://mattmahoney.net/dc/10gb.html
    Empty directories and file dates are not restored, but otherwise OK.
    Last edited by Matt Mahoney; 2nd June 2014 at 20:05.

  19. The Following User Says Thank You to Matt Mahoney For This Useful Post:

    Nania Francesco (2nd June 2014)

  20. #17
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Good news:
    - Added static Huffman bit coding only for match length with tables. Decompression is only 3-9 % slower than version 0.01!
    - in some cases the efficiency during decompression is remarkable!
    Last edited by Nania Francesco; 4th June 2014 at 01:22.

  21. #18
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    864
    Thanks
    46
    Thanked 104 Times in 82 Posts
    Do you consider implanting ECM prefiltering of cue+bin/ccd images ?
    and what about PCM extractor into PCM compressor for cue+bin/ccd as well ?

  22. #19
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    I released version is already a data type equal to that of ZCM, recognizes generically rle images, audio, and put into an ISO or TAR or other type of file not compressed. You can certainly improve but it's very interesting.
    LZA v.0.01 include 3 filter's for:
    - WAVE/PCM
    - BMP/PGM/PPM
    - RLE DATA
    Last edited by Nania Francesco; 5th June 2014 at 15:43.

  23. #20
    Member
    Join Date
    Jun 2013
    Location
    USA
    Posts
    98
    Thanks
    4
    Thanked 14 Times in 12 Posts
    Quote Originally Posted by SvenBent View Post
    Do you consider implanting ECM prefiltering of cue+bin/ccd images ?
    and what about PCM extractor into PCM compressor for cue+bin/ccd as well ?
    This sounds pretty useless. Why would you even have cue+bin files instead of mds+iso?

  24. #21
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    864
    Thanks
    46
    Thanked 104 Times in 82 Posts
    You are the one claiming its pretty useless please enlighten me why.

    but from my small gathering ecm would not work on a mds+iso as iso does not contain raw+96 sector in the first place but only the actual data layer,.
    Without the raw+96 info you cannot stored the wrongly ECC layers that are needed to backup certain games..

    If mds+iso does in fact store the raw+96 sectors then we are just debating what file format to put it in. never the less removing unneeded ECC data would still help compression


    -- note --
    actually i cant remember if cue/bin is able to store raw+96 or just raw and .CCD is needed to store raw+96. but i do believe mds+iso does not store raw at all.

  25. #22
    Member
    Join Date
    Jun 2013
    Location
    USA
    Posts
    98
    Thanks
    4
    Thanked 14 Times in 12 Posts
    IIRC, ecm works by removing the ECC data from the files and recalculating/reapplying it upon decompressing.

    ImgBurn (which is the only program that I use for optical media these days) will not allow you to back up a CD as cue/bin, only mds/iso. The only exception is audio CDs which have no ECC.

    Personally, I've never encountered these games you speak of, but storing bin files instead of iso sounds like a waste of space.

  26. #23
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    the latest news regarding the new version doesn't differ much about the latest updates to the data structure of the coder.
    The results are really interesting and in decompression better than option -cdp of Nanozip ( 5-10% in time of decompression and 5-10% in size )

  27. #24
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    864
    Thanks
    46
    Thanked 104 Times in 82 Posts
    Quote Originally Posted by Mangix View Post
    IIRC, ecm works by removing the ECC data from the files and recalculating/reapplying it upon decompressing.
    Exactly what i was saying. a RAW sector is a sector including the ECC part a RAW+96 is a sector including the ECC part and the 2 sub channels info as well
    This EEC data is is only for data CD and is unrelated to the C1 and C2 error correction layer on a CD.
    The c1 and C1 layer is also present on an audioCD

    Quote Originally Posted by Mangix View Post
    ImgBurn (which is the only program that I use for optical media these days) will not allow you to back up a CD as cue/bin, only mds/iso. The only exception is audio CDs which have no ECC.
    I dont know why the program you decide to use constitutes a augment for calling another format for useless. very rarely does a program target at single user.

    Quote Originally Posted by Mangix View Post
    Personally, I've never encountered these games you speak of,
    Again just because you never encountered them does not constitute a proof or even a valid argument they do not exist.
    Fake ECC errors are among the earliest adaption of copy protection and one of the reasons CUE/bin adapted being able to read raw sectors.
    The first many cd writers would read sector as data and thereby use the EEC data to correct a sector with was bad according to ECC and thereby "Fix" the sector
    The software on execution would read the files places on such bad sectors and see if the files was intact or "Fixed"

    Such a game among others is civilization 2 which is a game i treasure a lot.

    its funny you refuse to believe the existing of such copy protection when even the the creatoer of ECM infomr that ECM does does in fact keep fake ECC errors in copy protected images.

    What about copy protection? Don't they use bogus ECC data? Wouldn't ECM ruin that?

    ECM only eliminates EDC/ECC data for sectors where it's verifiable possible to recreate that data. If the ECC is bogus, ECM will preserve that bogus data.



    As extra info i a can inform as the next step many cd protections moved to mess around with the sub channels which was why it became a big fad with .CCD images which beside the raw sector a cue sheet also contained a file to store the sub channels. you have to make sure though you had read/write and software able to handle the raw+96 read and write mode ( data +ecc +96byts of sub channels)

    next step in line was the tried to make hard to write EFM patterns. EFM is Eight to Fourteen .. something which is a bit pattern to bit pattern remapping to avoid bit patterns that could make the laser focus lose track. BY abusing the EFM patters they could make it hard for burner to reproduce the correct EFM patterns and thereby corrupt the data.
    Copy protection today is more and more complex and newer images formats containing has been developed. like MDF/MDS
    all of these have a purpose off describing more and more information about a cd images, and just because its not useable to your usage pattern does not make them objectively useless.


    Quote Originally Posted by Mangix View Post
    but storing bin files instead of iso sounds like a waste of space.
    If its possible to store in Iso alone you are correct but for a lot of correct storing of CD images, ISO is useless (in the correct sense of the word)
    By your sense of useless iso/mdf is useless as well since the mdf file is unneeded and you can store an CD image in just an iso file.


    I hope in future you will be less inclined to let you lack of knowledge declare anything you don't know about as useless. it is a dangerous closeminded way to approach things.


    Back on topic.
    Im hoping to get some time to test LZA soon its looks like a promising project.
    Last edited by SvenBent; 26th June 2014 at 05:40.

  28. #25
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Released last version 0.10!

    - Added static Huffman coding for match !
    - Better compression (5- 20% )!
    - 5-20 % slower than old version !
    - LZ77 search core modified
    - best option extended to -mx5
    Attached Files Attached Files
    Last edited by Nania Francesco; 30th June 2014 at 11:43.

  29. The Following User Says Thank You to Nania Francesco For This Useful Post:

    Stephan Busch (1st July 2014)

  30. #26
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Result in Huge Files Compression Benchmark
    File Vm.dll size=4.244.176.896 bytes

    NANOZIP 0.07 -cd -m1.8g -> 1.173.141.528 bytes [benchmark results]

    with Intel Core i7 920 2.67 ghz 4GB ram:
    LZA v.0.10 -mx5 -b6 -h7 -> 1.179.091.707 bytes comp. 1154 sec. - decomp. 51 sec.
    NANOZIP 0.09 -cdp -m1.8g -t1 ->1.181.462.334 comp. 288 sec. - decomp. 100 sec.

    @ Matt
    Please , test LZA v.0.10 in LTCB and 10GB benchmarks!
    Last edited by Nania Francesco; 30th June 2014 at 22:00.

  31. #27
    Member Bloax's Avatar
    Join Date
    Feb 2013
    Location
    Dreamland
    Posts
    52
    Thanks
    11
    Thanked 2 Times in 2 Posts
    3.4 GB/s?
    Careful there, you might get a speeding ticket driving around like that!

  32. #28
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    I think he means 1154 sec (. and , are reversed in U.S. and European numbers).

    Anyway, test results show nice improvement:
    http://mattmahoney.net/dc/text.html#2508
    http://mattmahoney.net/dc/10gb.html

    But "lza.exe l 10gb.lza" crashes (read page fault) after completing the listing under Wine 1.6 in 64 bit Ubuntu.

  33. The Following User Says Thank You to Matt Mahoney For This Useful Post:

    Nania Francesco (30th June 2014)

  34. #29
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Thanks for all test Matt !
    As Matt has already noticed the LZA version use the enhancements to the structure of data analysis I've implemented with ZCM. It should not seem strange that less effective Flashzip LZA, but just doesn't have a structure so evolved. The future version of Flashzip should be able to support in principle the results of ZCM.

  35. The Following User Says Thank You to Nania Francesco For This Useful Post:

    Stephan Busch (1st July 2014)

  36. #30
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    873
    Thanks
    462
    Thanked 175 Times in 85 Posts
    When I use "lza a -r -s -v -mx5 -b7 -h7' the program tells me 'no memory'
    Can this be fixed? Or is an 64-bit executable available?

Page 1 of 7 123 ... LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •