Page 5 of 7 FirstFirst ... 34567 LastLast
Results 121 to 150 of 201

Thread: LZA archiver

  1. #121
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    678
    Thanks
    213
    Thanked 209 Times in 129 Posts
    Interesting - does it use two number state (Low, Range of range coding) or one number state (x of rANS) ?
    Is its decoding and encoding in the same or opposite direction?
    Last edited by Jarek; 12th November 2014 at 10:57.

  2. #122
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Difficult to answer, because the encoding and decoding is to the same direction but the inner core (I hope only for the moment) is that of RANS. The big difference is that the model is adaptive and can also be order1!
    Last edited by Nania Francesco; 12th November 2014 at 12:43.

  3. #123
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    678
    Thanks
    213
    Thanked 209 Times in 129 Posts
    Nania, as I have written, there is completely no problem making rANS or tANS/FSE adaptive e.g. order 1:
    go through the file like while using range coding, but just place (p_i, D_i) pairs to a buffer (D_i is identifier of probability distribution for this symbol, e.g. depending on the previous one).
    After that use rANS encoding in backward order, using D_i as the number of table for given step (e.g. table[D][x] instead of table[x]).
    Thanks of that, decoding is straightforward.

    Here are benchmarks of order 1 rANS of JamesB: http://encode.su/threads/1920-In-mem...ll=1#post37574
    Here is his implementation: http://encode.su/threads/2017-LzTurb...ll=1#post39920

  4. #124
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    But the code that you show me the frequency is not updated in real time, is calculated before not during compression and you're forced to write pre-calculated frequencies? This my version does not need! It's much easier the solution and not all that useless code!
    Excuse my understanding of English. But you want to make me understand that code like you have already created or give me suggestions to improve the program?
    Last edited by Nania Francesco; 12th November 2014 at 13:04.

  5. #125
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    678
    Thanks
    213
    Thanked 209 Times in 129 Posts
    In both range coding and rANS we need to quickly determine to which subrange (symbol) given position corresponds to - in rANS it is a bit simpler as the range is fixed ({0, ... , 2^n-1}), while in range coding it is varying.

    - the basic approach is just to define symbol[x] table, however updating it is costly.
    - perfect for updating would be using some binary search, however it would be costly for processing: log(alphabet size) instead of constant.
    - there is very nice Fabian's approach: alias method - we immediately determine one of identical size buckets, then perform one "if" to choose between the symbol corresponding to this bucket and "alias symbol" for this bucket - it can be easily undated e.g. by shifting dividers in buckets (here is a different implementation (Pascal Massimino): https://github.com/skal65535/fsc ).

    You want to say that you have a better approach?

  6. #126
    Member
    Join Date
    Jun 2009
    Location
    Kraków, Poland
    Posts
    1,474
    Thanks
    26
    Thanked 121 Times in 95 Posts
    Francesco:
    A in ANS or ABS stands for Asymmetric. If your coder encode and decode in the same direction then it's probably not ANS/ ABS but some variation of range coder. How your coder differs from range coder? Could you share some details? Do you use pair of variables ((low, range) or equivalently (low, high)) or one variable (as in ANS/ABS)?

  7. #127
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    @ Jarek
    I'm saying that my card is real time encoder range coder (like the old traditional range coder), adaptive (also in order 1) and does not need to write the frequencies. If this is enough to make it a my creation good, otherwise I will publish the code anyway because they will open source LZA!

    @ Piotr
    My code that is still undergoing improvement (I already have a working version of course) in the decompressor uses decompression of RANS but in compression uses a different approach. At the moment I don't know tell you more, expect Matt (and all ANS expert's )I confirm that my code is likely to be different in a way that I have the authorship, considering that is the first Range coder rans Adaptive.
    Attached Files Attached Files
    Last edited by Nania Francesco; 12th November 2014 at 14:01.

  8. #128
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    678
    Thanks
    213
    Thanked 209 Times in 129 Posts
    Quote Originally Posted by Nania Francesco View Post
    My code that is still undergoing improvement (I already have a working version of course) in the decompressor uses decompression of RANS but in compression uses a different approach.
    So you are saying that you have managed to make effective forward rANS encoding?
    I have tried to do something like that, but failed - can't wait to see it!

  9. #129
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Unfortunately I have a commitment and we hear tonight. I would appreciate your opinion about whether it can be considered a new creation? Also, since larger aid anyway I've given you along with Piotr and Matt, please help me understand what the correlation between the LowRange (old rc) and X and if there is a way to compute X in relation to the old LowRange in the same position naturally ?

  10. #130
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    678
    Thanks
    213
    Thanked 209 Times in 129 Posts
    If you indeed have some new contribution to ANS, I will definitely not forget about it.
    There is no simple correspondence between low/range of range coder and x of ANS - these methods are quite different ...

    Just take your time and present what you have when you are ready ...
    Last edited by Jarek; 12th November 2014 at 15:25.

  11. The Following User Says Thank You to Jarek For This Useful Post:

    Nania Francesco (12th November 2014)

  12. #131
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    Nania, I can't tell from .exe files. Did you mean to post source code?

  13. The Following User Says Thank You to Matt Mahoney For This Useful Post:

    Nania Francesco (12th November 2014)

  14. #132
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Hi Matt!
    Could you send me a private message on this site with your email working! The code remains protected at the moment!
    I'll send you the source code only to judge if it is to be considered a separate encoder than RANS (in my opinion static)
    Last edited by Nania Francesco; 13th November 2014 at 00:10.

  15. #133
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    My code is ready but I have not had the opportunity to be contacted by Matt.

  16. #134
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Matt you received the message?

    MSN gives me an error message "delivery status failure"

  17. #135
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 778 Times in 485 Posts
    Yes. The code from https://github.com/rygorous/ryg_rans says public domain so you should be OK to modify it.

  18. #136
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Ok thanks Matt !

  19. #137
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    Released last version 0.70 ! [nArANS]!

    - new core for better compression!
    - extended compression option to -m9 (-mx9)

    download from:
    http://heartofcomp.altervista.org/
    Last edited by Nania Francesco; 19th November 2014 at 13:35.

  20. #138
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    776
    Thanks
    63
    Thanked 270 Times in 190 Posts
    With both 0.70 versions I get crash like this:
    Unhandled exception at 0x0040762C in LZA.exe: 0xC0000005: Access violation reading location 0x114E9D6

  21. #139
    Member
    Join Date
    Oct 2011
    Location
    Leeds, West Yorkshire. UK.
    Posts
    2
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Same here on win7 x64

    Problem signature:
    Problem Event Name: APPCRASH
    Application Name: LZAx64.exe
    Application Version: 0.0.0.0
    Application Timestamp: 546c8235
    Fault Module Name: LZAx64.exe
    Fault Module Version: 0.0.0.0
    Fault Module Timestamp: 546c8235
    Exception Code: c0000005
    Exception Offset: 0000000000007331
    OS Version: 6.1.7600.2.0.0.256.1
    Locale ID: 1033
    Additional Information 1: c96e
    Additional Information 2: c96e28a692422f3d78a2bbdc6a619855
    Additional Information 3: eeeb
    Additional Information 4: eeeb3934d6cdb0a7fc2d2ab0f9945e44

  22. #140
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    With both 0.70 versions I get crash like this:
    Unhandled exception at 0x0040762C in LZA.exe: 0xC0000005: Access violation reading location 0x114E9D6
    Same here on win7 x64

    Problem signature:
    Problem Event Name: APPCRASH
    Application Name: LZAx64.exe
    Application Version: 0.0.0.0
    Application Timestamp: 546c8235
    Fault Module Name: LZAx64.exe
    Fault Module Version: 0.0.0.0
    Fault Module Timestamp: 546c8235
    Exception Code: c0000005
    Exception Offset: 0000000000007331
    OS Version: 6.1.7600.2.0.0.256.1
    Locale ID: 1033
    Additional Information 1: c96e
    Additional Information 2: c96e28a692422f3d78a2bbdc6a619855
    Additional Information 3: eeeb
    Additional Information 4: eeeb3934d6cdb0a7fc2d2ab0f9945e44
    Thanks for test !

    Released last version 0.70b ! [nArANS]!

    - corrected bugs in normal compression !

    download from:
    http://heartofcomp.altervista.org/

  23. The Following 3 Users Say Thank You to Nania Francesco For This Useful Post:

    Jarek (19th November 2014),Stephan Busch (19th November 2014),surfersat (19th November 2014)

  24. #141
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    919
    Thanks
    541
    Thanked 363 Times in 271 Posts
    Quote Originally Posted by Nania Francesco View Post
    Thanks for test !

    Released last version 0.70b ! [nArANS]!

    - corrected bugs in normal compression !

    download from:
    http://heartofcomp.altervista.org/
    I have a strange problem with this version on Win7 (it occurs in both 70b builds) - it compress only if I set option to all files in directory (lza.exe a -m9 test *), but when I specify particular file then there are an error "WARNING: No files Compressed xxxxxxx bytes to 104 bytes" where xxxxxxx is a proper lenght of file to be compressed.

    Darek

  25. #142
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    LZA file search core is "case sensitive". For example "c:\FP.LOG" is compressed and "c:\FP.log" not! LZA not support at moment UTF-16 file name !
    Last edited by Nania Francesco; 19th November 2014 at 16:33.

  26. #143
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    919
    Thanks
    541
    Thanked 363 Times in 271 Posts
    Quote Originally Posted by Nania Francesco View Post
    LZA file search core is "case sensitive". For example "c:\FP.LOG" is compressed and "c:\FP.log" not!
    ok, now it works!

  27. The Following User Says Thank You to Darek For This Useful Post:

    Nania Francesco (19th November 2014)

  28. #144
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    LZA v.070 gain 1^ and 2^ position in decompression efficiency (WCC 2014) !

  29. #145
    Member
    Join Date
    Sep 2011
    Location
    uk
    Posts
    237
    Thanks
    187
    Thanked 16 Times in 11 Posts
    Few comments, latest b version

    1) case sensitive stuff in windows is confusing so if it is still included pls mention it in help (or preferably remove!)
    2) tried latest version 32 bit and 64 bit on win 7. On 64 bit version get 'lza not working' error message from o/s - assume this is a memory problem. Suggest this is checked within lza and an error message 'not enough memory' output. 32 bit version works.
    3) suggest that in the help the switches have - added eg -r Recursive directories etc.
    4) some sort of % progress output or anyway some indication of progress, eg as for arc
    5) performance, using 32 bit version
    lza a arch file, compressed 4G jessie .img fto 1.5G in 820 seconds elapsed on fx4200 4 cores. Does not use 100% of 4 cores
    arc a arch file, same file compresssed to 1.3G. Uses 100% of 4 cores 800 seconds cpu time, 230 seconds elapsed
    6) is there an option -mx to use all of the cores? If so mention in help.

    John

  30. #146
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    LZA at moment is single core version!

  31. #147
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    10GB benchmark results of LZA v.070b
    System 7 : Intel Core i7 920, 2.66 GHz, 4+4 HT, 6 GB, Win7. Tested by Nania Francesco.

    C:\>timer lza a -r -s -mx9 -b7 -h9 c:\pro.lza g:\test\10gb\
    LZA archiver v.0.70b. Warning is demo version.

    Copyright (C) 2014 NANIA Francesco A.

    Archive is c:\pro.lza
    Compressed 10000000000 bytes to 3108986473 bytes


    Kernel Time = 44.756 = 0%
    User Time = 6503.354 = 94%
    Process Time = 6548.110 = 95% Virtual Memory = 4096 MB
    Global Time = 6864.451 = 100% Physical Memory = 4096 MB

    C:\>timer lza x c:\pro.lza g:\proco\
    LZA archiver v.0.70b. Warning is demo version.

    Copyright (C) 2014 NANIA Francesco A.

    ARGV=g:\proco\
    Archive=c:\pro.lza Wild=* DEST=g:\proco\\
    OK!

    Exit code: 1


    Kernel Time = 33.493 = 8%
    User Time = 85.660 = 21%
    Process Time = 119.153 = 29% Virtual Memory = 1077 MB
    Global Time = 377.840 = 100% Physical Memory = 1070 MB
    Last edited by Nania Francesco; 20th November 2014 at 18:53.

  32. #148
    Member
    Join Date
    Sep 2011
    Location
    uk
    Posts
    237
    Thanks
    187
    Thanked 16 Times in 11 Posts
    Quote Originally Posted by Nania Francesco View Post
    LZA at moment is single core version!
    eta for multi core? what's roadmap for lza? John

  33. #149
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    As for multicore is difficult to give an answer. I will try to find a solution or opt to do insert LZA in 7ZIP package maybe; the match should be compatible with 7ZIP with some small change. I have not yet contacted Igor Pavlov but I will just like attempt. For the second question I just see where squeeze still the core remaining at a high speed of decompression!

  34. The Following User Says Thank You to Nania Francesco For This Useful Post:

    avitar (20th November 2014)

  35. #150
    Member
    Join Date
    Sep 2011
    Location
    uk
    Posts
    237
    Thanks
    187
    Thanked 16 Times in 11 Posts
    Quote Originally Posted by Nania Francesco View Post
    As for multicore is difficult to give an answer. I will try to find a solution or opt to do insert LZA in 7ZIP package maybe; the match should be compatible with 7ZIP with some small change. I have not yet contacted Igor Pavlov but I will just like attempt. For the second question I just see where squeeze still the core remaining at a high speed of decompression!
    IMO, no multicore is the biggest issue making it difficult to use lza as serious archiver & get people using it As you know most processors now, even on laptops have n cores, so you are giving lza a huge factor approx. n disadvantage cf arc, zpaq etc, which is much more than any the advantage in clever schemes you may have!

    Progress indication is next most important one (from my list in #145), especially if you don't have multi core because it is difficult to decide whether it'll finish in few minutes, or there is time to make tea/go to sleep etc. The others are very easy to implement.

    John

Page 5 of 7 FirstFirst ... 34567 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •