Page 2 of 7 FirstFirst 1234 ... LastLast
Results 31 to 60 of 201

Thread: LZA archiver

  1. #31
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Hi Stephan!

    I have no great experience with x 64 versions. Should I experiment and read some guide but of course I'll release a version which supports 2 GB memory

  2. #32
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    873
    Thanks
    462
    Thanked 175 Times in 85 Posts
    Hi Francesco

    In my opinion 64-bit offers speed increasement and is worth a try

  3. The Following User Says Thank You to Stephan Busch For This Useful Post:

    Nania Francesco (3rd July 2014)

  4. #33
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    This first version release 0.10 x64 compatible (compiled with GCC 4.8.1 x64) , at least on my computer did not give any problems. You can use the option-b7-h7 simultaneously!
    Let me know if you notice problems

    LTCB:

    C:\>timer lza a -r -s -v -mx5 -b7 -h7 c:\pro.lzt d:\test\enwik9

    Timer 3.01 Copyright (c) 2002-2003 Igor Pavlov 2003-07-10
    LZA archiver v.0.10. Warning is demo version.

    copyright (C) 2014 By Nania Francesco Antonio

    Archive is c:\pro.lzt
    compress.. d:/test/enwik9 1000000000 to 246671317
    Compressed 1000000000 bytes to 246671317 bytes

    Kernel Time = 1.482 = 00:00:01.482 = 0%
    User Time = 252.628 = 00:04:12.628 = 98%
    Process Time = 254.110 = 00:04:14.110 = 98%
    Global Time = 257.745 = 00:04:17.745 = 100%

    C:\>timer lza x c:\pro.lzt c:\proco\

    Timer 3.01 Copyright (c) 2002-2003 Igor Pavlov 2003-07-10
    LZA archiver v.0.10. Warning is demo version.

    copyright (C) 2014 By Nania Francesco Antonio

    ARGV=c:\proco\
    Archive=c:\pro.lzt Wild=* DEST=c:\proco\\
    OK!

    Kernel Time = 0.858 = 00:00:00.858 = 8%
    User Time = 9.531 = 00:00:09.531 = 90%
    Process Time = 10.389 = 00:00:10.389 = 98%
    Global Time = 10.514 = 00:00:10.514 = 100%
    HFCB: Huge Files Compression Benchmark
    C:\>timer lza a -r -s -v -mx5 -b7 -h7 c:\pro.lzt d:\test\vm.dll
    Timer 3.01 Copyright (c) 2002-2003 Igor Pavlov 2003-07-10
    LZA archiver v.0.10. Warning is demo version.

    copyright (C) 2014 By Nania Francesco Antonio
    Archive is c:\pro.lzt
    compress.. d:/test/vm.dll 4244176896 to 1174087337
    Compressed -50790400 bytes to 1174087337 bytes

    Kernel Time = 4.758 = 00:00:04.758 = 0%
    User Time = 1160.054 = 00:19:20.054 = 98%
    Process Time = 1164.812 = 00:19:24.812 = 98%
    Global Time = 1181.848 = 00:19:41.848 = 100%

    LZA 0.10 32Bit Version -> 1.179.091.707 bytes
    Attached Files Attached Files
    Last edited by Nania Francesco; 4th July 2014 at 09:38.

  5. #34
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts

  6. The Following User Says Thank You to Matt Mahoney For This Useful Post:

    Nania Francesco (6th July 2014)

  7. #35
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    The transformation of the LZA in a full of range encoder is completed. I wanted to know if there is a way to weigh the cost of the bits in a range coder like that used by QUAD and TORNADO is compatible with existing uses such as 7ZIP or CSC32.
    for example:

    void Encode(Rangecounter &counter, int s)
    {
    x1 += (counter.cum[s] * (x2/=counter.total));
    x2 *= counter.cnt[s];
    while (x2 <0x1000000)
    {
    ( x2 <<= 8 );ShiftLow();
    }
    counter.Add(s);
    }

    The cost of symbol s is ?

    in comparison to standard bit encoder:

    void SymbolbitEncode( int s)
    {
    encode(s&128 );
    encode(s&64);
    encode(s&32);
    encode(s&16);
    encode(s&8 );
    encode(s&4);
    encode(s&2);
    encode(s&1);
    }
    Last edited by Nania Francesco; 30th August 2014 at 21:25.

  8. #36
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,474
    Thanks
    26
    Thanked 121 Times in 95 Posts
    The cost of symbol when arithmetic coding is the opposite of logarithm of its probability. You probably may want to use some rough approximation so you can speed things up. Look here:
    http://encode.su/threads/1449-Adapti...ll=1#post27862
    In lglut.h is a code for computation of logarithm base2. ilog computes integer logarithm, while nlog computes a fixed point logarithm with some digits after the point.

    Remember that log(a / b) = log(a) - log(b) so you can use two lookups and subtraction instead of division and one lookup. So if a symbol has probability (counter.cnt[s] / counter.total) then you compute (log(counter.total) - log(counter.cnt[s])) (remember the negation) and you have symbol length (cost).

  9. The Following User Says Thank You to Piotr Tarsa For This Useful Post:

    Nania Francesco (30th August 2014)

  10. #37
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Price of range coding:

    for (int i=0;i<(4096>>3);i++)
    {
    BitLog[i]=(u32)(128*log((float)(i*8+4)/4096)/log(0.5));
    }

    for (int i=0;i<(8192>>3);i++)
    {
    RangeLog[i]=(u32)(256*log((float)(i*8+4)/8192)/log(0.5));
    }

    For Bit encoding:

    int Testbit(bool y,short &p)
    {
    if (y)
    return BitLog[p>>3];
    else
    return BitLog[(4031-p)>>3];
    }

    For symbol range encoding;

    int SymbolCost(Rangecounter &counter, int s)
    {
    return RangeLog[(counter.total-counter.cnt[s])>>3]; or return RangeLog[counter.total>>3]-RangeLog[counter.cnt[s]>>3]; ?
    }
    Last edited by Nania Francesco; 31st August 2014 at 15:07.

  11. #38
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,474
    Thanks
    26
    Thanked 121 Times in 95 Posts
    Code:
    int SymbolCost(Rangecounter &counter, int s)
    {
    return RangeLog[(counter.total-counter.cnt[s])>>3]; or return RangeLog[counter.total>>3]-RangeLog[counter.cnt[s]>>3]; ? 
    }
    The latter form is correct (if you compute the LUT right).

    If you sum the costs of each symbol and flag in file you should get the size of compressed file in bits. That is an ultimate test of correctness

  12. #39
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    682
    Thanks
    215
    Thanked 213 Times in 131 Posts
    Why don't you use some ANS instead? While providing similar compression rate, for large alphabet it has a few times faster encoding and up to ~9 times faster decoding: http://encode.su/threads/1920-In-mem...entropy-coders

  13. #40
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    864
    Thanks
    46
    Thanked 104 Times in 82 Posts
    some issues found. i did
    LZA_x64.exe a -mx5 -h7 -b7 test.lza *.bin
    but it only compressed one of the 3 .bin files in the folder (estimated from the size info after compression).

    then i did
    LZA_x64.exe l test.bin
    and it keept on repeating the same file name over and over and over and i had to ctrl+c to break it

    Trying to decompress with
    LZA_x64.exe c test.bin
    and it crashed with a windows error


    the 3 files where quake1 + 2 expansion cd images

    03/25/2014 03:59 335,853,840 ARMAGON.bin
    03/25/2014 04:09 349,963,488 DISSOLUTION.bin
    03/29/2014 20:18 662,734,800 QUAKE101.BIN
    Last edited by SvenBent; 31st August 2014 at 20:30.

  14. The Following User Says Thank You to SvenBent For This Useful Post:

    Nania Francesco (31st August 2014)

  15. #41
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Released last version 0.50!

    - LZ77 search core modified
    - Compression engine is based on new fast Symbol Range coder + Standard bit range coder + fast bit coder ... for better compression!

    download from:
    http://heartofcomp.altervista.org/
    Last edited by Nania Francesco; 6th September 2014 at 18:05.

  16. #42
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    468
    Thanks
    203
    Thanked 81 Times in 61 Posts
    Thanks! Tried to test, but its does nothing in my PC. Previous version worked fine...
    Machine: XP 32 bits SP3 1g ram 1.6ghz single core amd Sempron 2800+

    Exit Code: 0
    Last edited by Gonzalo; 6th September 2014 at 22:38.

  17. #43
    Member
    Join Date
    Sep 2011
    Location
    uk
    Posts
    237
    Thanks
    187
    Thanked 16 Times in 11 Posts
    On my pc (similar to gonzalo's) it says cant find a dll libgcc-s dw2 or something. j

  18. #44
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    873
    Thanks
    462
    Thanked 175 Times in 85 Posts
    doesn't work on my system either..
    the executable requires libgcc_s_dw2-1.dll and if it is in the same folder,
    an error message appears - application could not be started (error 0xc000007b)

  19. #45
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Released last version 0.50!

    - LZ77 search core modified
    - Compression engine is based on new fast Symbol Range coder + Standard bit range coder + fast bit coder ... for better compression!
    .........
    New compiled version 0.50 is online !

    This happened since I first installed the latest version of GCC and I realized that requires additional DLLs. This release is compiled with GCC 4.8.1 and seems to not require DLL!

    download from:
    http://heartofcomp.altervista.org/
    Last edited by Nania Francesco; 7th September 2014 at 02:32.

  20. #46
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    468
    Thanks
    203
    Thanked 81 Times in 61 Posts
    Solved! It was that.

  21. #47
    Member
    Join Date
    Sep 2011
    Location
    uk
    Posts
    237
    Thanks
    187
    Thanked 16 Times in 11 Posts
    Well I don't get the missing dll message now with the latest download, but running lza at command line just does nothing. Not sure how to get the help. Is it --help? or /? or what? Nothing works - if I type random characters as parameters! Then get an 'xp has failed error messahe do you want to send' message.

    xp sp3 1.5G memory, 1 core. Sounds similar to gonzalo's.

    J

  22. #48
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    This version is 32-bit executable, 64-bit release version I hope tomorrow!
    In my various tests showed levels of LZHAM but under compression is 7 times faster whit only 1 core !
    Last edited by Nania Francesco; 7th September 2014 at 02:56.

  23. #49
    Member
    Join Date
    May 2012
    Location
    United States
    Posts
    323
    Thanks
    178
    Thanked 52 Times in 37 Posts
    Quote Originally Posted by avitar View Post
    Well I don't get the missing dll message now with the latest download, but running lza at command line just does nothing. Not sure how to get the help. Is it --help? or /? or what? Nothing works - if I type random characters as parameters! Then get an 'xp has failed error messahe do you want to send' message.

    xp sp3 1.5G memory, 1 core. Sounds similar to gonzalo's.

    J
    I am also having this exact same problem

  24. #50
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    This version is compiled with GCC option: -O3 -ffast-math -march=i686 -mtune=i686 for generic CPU (slow).

    __________________________________________________ ______________________________________
    HFCB: Huge Files Compression Benchmark :with Intel Core i7 920 2.67 ghz 6GB ram:
    option : a -r -s -v -mx5 -b7 -h6
    LZA compress.. d:/TEST/vm.dll 4244176896 to 1071931845
    Kernel Time = 2.823 = 00:00:02.823 = 0%
    User Time = 998.031 = 00:16:38.031 = 97%
    Process Time = 1000.855 = 00:16:40.855 = 98%
    Global Time = 1020.371 = 00:17:00.371 = 100%

    Decompression time

    Kernel Time = 2.480 = 00:00:02.480 = 4%
    User Time = 34.257 = 00:00:34.257 = 67%
    Process Time = 36.738 = 00:00:36.738 = 72%
    Global Time = 50.575 = 00:00:50.575 = 100%

    WINRAR 5.10 (option -mt3 -m5 -ma5 -md512m) -> 1.046.052.647 bytes (decompression in 51,340 sec.)
    Attached Files Attached Files
    Last edited by Nania Francesco; 9th September 2014 at 01:48.

  25. #51
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    150
    Thanks
    30
    Thanked 59 Times in 35 Posts
    Good job!
    i686 version crashes for enwik7.
    lza a -mx5 out enwik7


    normal version is always crash...

  26. #52
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Released last version 0.50b!
    For old version 0.50: Actually I tried to launch the program with a core 2 duo and went into crush. It is strange that with the new GCC compilers the memory is so sensitive.

    - This version is (version 0.50) compiled with old GCC 4.6.1 (stable with all cpu)

    download from:
    http://heartofcomp.altervista.org/
    Last edited by Nania Francesco; 8th September 2014 at 00:44.

  27. #53
    Member
    Join Date
    Feb 2013
    Location
    San Diego
    Posts
    1,057
    Thanks
    54
    Thanked 71 Times in 55 Posts
    I've read that new compilers are becoming more aggressive about optimizations and this is breaking old code that depends on parts of the language that are undefined having specific results.

  28. #54
    Member
    Join Date
    Sep 2011
    Location
    uk
    Posts
    237
    Thanks
    187
    Thanked 16 Times in 11 Posts
    Latest version downloaded - still broken here. running from cl nothing happens then after ctrl/c a M$ error screen saying 'lza.exe has encountered problems and needs to close', please tell M$. I'll try one more 'new' version, then I'll have to give up! J

  29. #55
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Released last version 0.51!

    - This version is more stable ! I hope this time they have corrected the error. Just to clarify this version and 0.50b does not use filters for exe and txt (removed)

    download from:
    http://heartofcomp.altervista.org/
    Last edited by Nania Francesco; 8th September 2014 at 20:12.

  30. The Following User Says Thank You to Nania Francesco For This Useful Post:

    Stephan Busch (8th September 2014)

  31. #56
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    873
    Thanks
    462
    Thanked 175 Times in 85 Posts
    Will there be a 64-bit compile of LZA again?
    Why are exe and txt filters removed?
    Will future versions have those filters again?

    Have you tried rANS and/or tANS as entropy coder?

  32. #57
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Surely I'll release a 64 bit version, but this time I want to improve the code allowing you to use 4 GB of buffers for the match. I removed the exe and txt filters as I deal in solid mode, while improving the compression of individual files, changing the overall structure of the buffer, does not allow a long match and especially slows down file decompression of 5-10% at times. If I'll find a filter exe and txt fast and able to leave the structure almost intact will put them. Consider then that Stephan with version 1.0 as promised I will make free code and I do not want some someone whining about code too similar. As regards ANS and RANS sincerely before I would understand how it works although I can allow my experimental Rangecoder (other than but similar operation Bulat) is definitely higher in the ratio even if slower decompression (specifically is about three times faster than a bit coder like fpaq0).

    @Avitar and Xezz
    LZA running now?
    Last edited by Nania Francesco; 9th September 2014 at 01:57.

  33. #58
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    682
    Thanks
    215
    Thanked 213 Times in 131 Posts
    Regarding compression rate, both tANS and rANS can work as close theoretical maximum (Shannon entropy) as you want.
    Regarding understanding, the basic ANS concept is really simple:
    - spread symbols: assign a symbol to each natural number in more or less uniform way with p[s] densities (a finite range like {L, ..., 2L-1} in practice),
    - enumerate symbol appearances,
    - now the encoding step is: to add information from symbol s to information in state/number x, go to x-th appearance of s (in analogy to x -> 2x + s in standard binary system).
    Now add renormalization to stay in {L, ..., 2L-1} range and you have tANS, like Yann's FSE.
    Mixing different probability distributions, also binary ones (speed like for arithmetic coding), is not a problem.

    How much time LZA spends on entropy decoding? You can easily divide it by 2 or 3 ... leaving LZHAM behind.
    Last edited by Jarek; 9th September 2014 at 08:47.

  34. The Following User Says Thank You to Jarek For This Useful Post:

    Nania Francesco (9th September 2014)

  35. #59
    Member
    Join Date
    Dec 2012
    Location
    japan
    Posts
    150
    Thanks
    30
    Thanked 59 Times in 35 Posts
    It works!


    enwik7 compression
    switch result
    nothing 4172503
    -m1 3634685
    -m2 3437545
    -m3 3371425
    -m4 3423649
    -m5 3619842
    -mx1 3560970
    -mx2 3354197
    -mx3 3285542
    -mx4 3332449
    -mx5 3525578


    enwik8 compression
    m3 32753542
    m4 33125755


    Why higher is worse?

  36. #60
    Member
    Join Date
    Sep 2011
    Location
    uk
    Posts
    237
    Thanks
    187
    Thanked 16 Times in 11 Posts
    nope, v0.51 still broken here, same as my comment, so giving up until someone says it is ok. Hard to believe that it is that hard to make it work on old hardware, most other compressors here do! Think you don't test it eg on xp?

    xezz - what command line exactly do you use?

    J

Page 2 of 7 FirstFirst 1234 ... LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •