Page 19 of 19 FirstFirst ... 9171819
Results 541 to 555 of 555

Thread: FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)

  1. #541
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    yes. why you need it?
    Last edited by Bulat Ziganshin; 20th November 2015 at 17:41.

  2. #542
    Member Dimitri's Avatar
    Join Date
    Nov 2015
    Location
    Greece
    Posts
    48
    Thanks
    21
    Thanked 30 Times in 14 Posts
    Bulat, are you planning on using lzma2 stdin-to-stdout at some point ??? maybe fazip with lzma2 stdin/stdout

  3. #543
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    You missed the party)
    4x4:lzma exists for years in both FreeArc and FAZip. And yes, with stdin\out too in FAZip. And yes, lzma2 is worser then classic lzma due limitation of summary value of LP+LC value.
    So everything already implemented.
    Last edited by Skymmer; 22nd November 2015 at 03:05.

  4. #544
    Member Dimitri's Avatar
    Join Date
    Nov 2015
    Location
    Greece
    Posts
    48
    Thanks
    21
    Thanked 30 Times in 14 Posts
    maybe worse but sometimes ratio isnt everything .... how about speed ?? :P

  5. #545
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    681
    Thanks
    37
    Thanked 168 Times in 84 Posts
    4x4:lzma implements the same as lzma2. Just try it )

  6. #546
    Member Dimitri's Avatar
    Join Date
    Nov 2015
    Location
    Greece
    Posts
    48
    Thanks
    21
    Thanked 30 Times in 14 Posts
    will try to. btw thanks above all i love experimenting
    Last edited by Dimitri; 22nd November 2015 at 05:01.

  7. #547
    Member
    Join Date
    Nov 2015
    Location
    congo
    Posts
    5
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    yes. why you need it?
    i use FAZIP instead of XZ or TAR to transfer files over slow wan link.
    Last edited by steom; 23rd November 2015 at 09:33.

  8. #548
    Member
    Join Date
    Mar 2018
    Location
    sun
    Posts
    34
    Thanks
    19
    Thanked 15 Times in 7 Posts
    Greetings Bulat, I use srep 64bit v3.93a with latest FA 0.667. Srep works, however when I stress computer during compression it tend to corrupt data and during decompression I get srep error. For example when I mine crypto during time I compress(even on GPU only) or run something that also load CPU too much.

    Originally I thought its my computer problem but its not, computer is stable and I since I upgraded to different one - with same issues. And this dont happen to other tools like ztool/precomp or external lzma. Only with srep. I can stress computer if I exclude srep and it compress fine. And if I dont stress computer during compression with srep it will also compress fine - without crc error when decompressing.

    Clearly something is wrong with srep during CPU hangouts, do you maybe have an idea or advice what to try, which settings etc? I use it with -m3f:mem2g. Could accelerator or some other feature have to do with this? It never happened with older 32bit srep for example. I also have a source code so perhaps you could point me where to change some values and recompile if it helps(also could compiler optimization be a culprit)? (I use win7 64bit btw.)

    Thank you, spasiba.

  9. The Following 2 Users Say Thank You to elit For This Useful Post:

    78372 (13th April 2018),kassane (13th April 2018)

  10. #549
    Member
    Join Date
    May 2017
    Location
    Hungary
    Posts
    6
    Thanks
    39
    Thanked 3 Times in 3 Posts
    i have the same issue !

  11. The Following User Says Thank You to hunman For This Useful Post:

    78372 (13th April 2018)

  12. #550
    Member
    Join Date
    Mar 2018
    Location
    sun
    Posts
    34
    Thanks
    19
    Thanked 15 Times in 7 Posts
    I think I fixed that srep issue. I need more proof but so far I was able to pack 40gb while mining on both CPU and GPU, which before meant almost certain data corruption. Code itself remain unchanged(its latest public srep v393a) but I recompiled it with no -O optimization minus some other flags and on latest gcc/msys2. Specifically I used this(in msys2 console):

    Code:
    sources="Common/Common.cpp srep.cpp"
    options="-DFREEARC_WIN -DFREEARC_INTEL_BYTE_ORDER -DUNICODE -D_UNICODE -ICommon -IEncryption -IEncryption/headers -IEncryption/hashes $sources"
    options_gcc4="$options -march=native -Wno-write-strings -Wno-unused-result -Wno-attributes -Wformat -lole32 -luuid -static"
    
    g++ $options_gcc4 -osrep64g.exe -m64 -DLTC_NO_ASM $*

    I also compiled one as -mtune=native -msse2 for universal CPU's usage. Note -DLTC_NO_ASM was needed or it would not compile(ROLc boundary errors..), but has unlikely to do with the issue.

    Pls let me know if this one helped with data corruption during compression or not. Preferably also use external LZMA, like xnlz or 7zip/xz - just in case.
    Attached Files Attached Files

  13. The Following 4 Users Say Thank You to elit For This Useful Post:

    78372 (13th April 2018),hunman (13th April 2018),kassane (13th April 2018),Simorq (14th April 2018)

  14. #551
    Member
    Join Date
    Mar 2018
    Location
    sun
    Posts
    34
    Thanks
    19
    Thanked 15 Times in 7 Posts
    Unfortunately, issue still remain, albeit it does seems to me occurrence is less frequent. This night I packed another 17g data, one with srep+xnlz and another with srep+lolz. First one tested without error but on another one I got srep error again. Therefore, I consider this issue NOT fixed. Please ignore this thread/links for now unlit real 100% solution is found. I read elsewhere that increasing srep's block size may help so I will try that for some period and see if it help.

  15. The Following User Says Thank You to elit For This Useful Post:

    hunman (14th April 2018)

  16. #552
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    I now work on new SREP version. I confirm that SREP 3.9x had bugs, which i fixed now. I recommend to use SREP 3.2

  17. The Following 9 Users Say Thank You to Bulat Ziganshin For This Useful Post:

    78372 (17th May 2018),elit (31st May 2018),hunman (19th May 2018),Mike (17th May 2018),oltjon (17th May 2018),Sergey3695 (17th May 2018),Simorq (22nd May 2018),slipstream (17th May 2018),Zonder (18th May 2018)

  18. #553
    Member
    Join Date
    Mar 2018
    Location
    sun
    Posts
    34
    Thanks
    19
    Thanked 15 Times in 7 Posts
    Big thanks Bulat, but I need to ask you, will you still release new Srep as a free and/or open source project or will it only be part of your next proprietary FA?

  19. #554
    Member
    Join Date
    Mar 2018
    Location
    sun
    Posts
    34
    Thanks
    19
    Thanked 15 Times in 7 Posts
    Greetings Bulat, sorry I wasn't here for long but since you wrote back then:
    I confirm that SREP 3.9x had bugs, which i fixed now.
    Well, so where is it? Can't find the link...
    Thanks.

  20. #555
    Member
    Join Date
    Mar 2018
    Location
    sun
    Posts
    34
    Thanks
    19
    Thanked 15 Times in 7 Posts
    I narrowed it. This time I believe almost for sure as I tested multiple times and could repeat it. The problem seems future-lz, or m3f - m5f. If you use only m3 - m5 you should never get that error. Then the problem is that you cannot decompress from stdin in one go(without future_lz), it will have to first decompress (lzma..) and then separately srep. If you have enough RAM, -m0 with big dict. should work without even touching disk and I also stress-tested it and seem stable as well. I am already looking at the code but I dont see anything yet so not sure if I can ever fix it. The fact that this bug happen only if you stress CPU in parallel should mean code itself should be fine, looks more like lib/threading bug or something.
    Bulat, cut the crap already and tell me at least where to look . Right now I suspect something in this range(srep.cpp):

    Code:
    if (FUTURE_LZ)
            {
              // Read next input block
              checked_file_read (ftemp, buf, block->size);
    
              // Copy literal data, not covered by LZ matches, to outbuf[]
              char *in = buf,  *out = outbuf;
              for (STAT *stat = block->statbuf;  stat < block->statend; )
              {
                DECODE_LZ_MATCH(stat, false, ROUND_MATCHES, BASE_LEN, 0,  lit_len, LZ_MATCH, lz_match);
                memcpy(out, in, lit_len);
                out += lit_len;
                in  += lit_len + lz_match.len;
              }
              memcpy(out, in, block->size - (in-buf));
              out += block->size - (in-buf);
    
              // Write compressed block literals to compressed file
              checked_file_write (fout, outbuf, out-outbuf);
    Code:
    if (FUTURE_LZ)
            {
              block->header[2] = stat_size;
              checked_file_write (fout, block->header, header_size);
            }

Page 19 of 19 FirstFirst ... 9171819

Similar Threads

  1. Dict preprocessor
    By pat357 in forum Data Compression
    Replies: 5
    Last Post: 2nd May 2014, 21:51
  2. Wav Compression in Freearc, from TTA to TAK ?
    By eleria in forum Data Compression
    Replies: 10
    Last Post: 4th February 2010, 21:14
  3. REP and Delta fails with big files
    By SvenBent in forum Data Compression
    Replies: 14
    Last Post: 23rd November 2008, 19:41
  4. 4x4 bug?
    By m^2 in forum Data Compression
    Replies: 11
    Last Post: 15th November 2008, 19:25
  5. 4x4 - multithreaded compressor
    By Bulat Ziganshin in forum Forum Archive
    Replies: 12
    Last Post: 19th April 2008, 17:25

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •