FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)

[Bulat Ziganshin]

yes. why you need it?

[Dimitri]

Bulat, are you planning on using lzma2 stdin-to-stdout at some point ??? maybe fazip with lzma2 stdin/stdout

[Skymmer]

You missed the party)
4x4:lzma exists for years in both FreeArc and FAZip. And yes, with stdin\out too in FAZip. And yes, lzma2 is worser then classic lzma due limitation of summary value of LP+LC value.
So everything already implemented.

[Dimitri]

maybe worse but sometimes ratio isnt everything .... how about speed ?? :P

[Skymmer]

4x4:lzma implements the same as lzma2. Just try it )

[Dimitri]

will try to. btw thanks above all i love experimenting

[steom]

yes. why you need it?

i use FAZIP instead of XZ or TAR to transfer files over slow wan link.

[elit]

Greetings Bulat, I use srep 64bit v3.93a with latest FA 0.667. Srep works, however when I stress computer during compression it tend to corrupt data and during decompression I get srep error. For example when I mine crypto during time I compress(even on GPU only) or run something that also load CPU too much.

Originally I thought its my computer problem but its not, computer is stable and I since I upgraded to different one - with same issues. And this dont happen to other tools like ztool/precomp or external lzma. Only with srep. I can stress computer if I exclude srep and it compress fine. And if I dont stress computer during compression with srep it will also compress fine - without crc error when decompressing.

Clearly something is wrong with srep during CPU hangouts, do you maybe have an idea or advice what to try, which settings etc? I use it with -m3f:mem2g. Could accelerator or some other feature have to do with this? It never happened with older 32bit srep for example. I also have a source code so perhaps you could point me where to change some values and recompile if it helps(also could compiler optimization be a culprit)? (I use win7 64bit btw.)

Thank you, spasiba.

[hunman]

i have the same issue !

[elit]

I think I fixed that srep issue. I need more proof but so far I was able to pack 40gb while mining on both CPU and GPU, which before meant almost certain data corruption. Code itself remain unchanged(its latest public srep v393a) but I recompiled it with no -O optimization minus some other flags and on latest gcc/msys2. Specifically I used this(in msys2 console):

sources="Common/Common.cpp srep.cpp"
options="-DFREEARC_WIN -DFREEARC_INTEL_BYTE_ORDER -DUNICODE -D_UNICODE -ICommon -IEncryption -IEncryption/headers -IEncryption/hashes $sources"
options_gcc4="$options -march=native -Wno-write-strings -Wno-unused-result -Wno-attributes -Wformat -lole32 -luuid -static"

g++ $options_gcc4 -osrep64g.exe -m64 -DLTC_NO_ASM $*

I also compiled one as -mtune=native -msse2 for universal CPU's usage. Note -DLTC_NO_ASM was needed or it would not compile(ROLc boundary errors..), but has unlikely to do with the issue.

Pls let me know if this one helped with data corruption during compression or not. Preferably also use external LZMA, like xnlz or 7zip/xz - just in case.

[elit]

Unfortunately, issue still remain, albeit it does seems to me occurrence is less frequent. This night I packed another 17g data, one with srep+xnlz and another with srep+lolz. First one tested without error but on another one I got srep error again. Therefore, I consider this issue NOT fixed. Please ignore this thread/links for now unlit real 100% solution is found. I read elsewhere that increasing srep's block size may help so I will try that for some period and see if it help.

[Bulat Ziganshin]

I now work on new SREP version. I confirm that SREP 3.9x had bugs, which i fixed now. I recommend to use SREP 3.2

[elit]

Big thanks Bulat, but I need to ask you, will you still release new Srep as a free and/or open source project or will it only be part of your next proprietary FA?

[elit]

Greetings Bulat, sorry I wasn't here for long but since you wrote back then:

I confirm that SREP 3.9x had bugs, which i fixed now.

Well, so where is it? Can't find the link...
Thanks.

[elit]

I narrowed it. This time I believe almost for sure as I tested multiple times and could repeat it. The problem seems future-lz, or m3f - m5f. If you use only m3 - m5 you should never get that error. Then the problem is that you cannot decompress from stdin in one go(without future_lz), it will have to first decompress (lzma..) and then separately srep. If you have enough RAM, -m0 with big dict. should work without even touching disk and I also stress-tested it and seem stable as well. I am already looking at the code but I dont see anything yet so not sure if I can ever fix it. The fact that this bug happen only if you stress CPU in parallel should mean code itself should be fine, looks more like lib/threading bug or something.
Bulat, cut the crap already and tell me at least where to look . Right now I suspect something in this range(srep.cpp):

if (FUTURE_LZ)
        {
          // Read next input block
          checked_file_read (ftemp, buf, block->size);

          // Copy literal data, not covered by LZ matches, to outbuf[]
          char *in = buf,  *out = outbuf;
          for (STAT *stat = block->statbuf;  stat < block->statend; )
          {
            DECODE_LZ_MATCH(stat, false, ROUND_MATCHES, BASE_LEN, 0,  lit_len, LZ_MATCH, lz_match);
            memcpy(out, in, lit_len);
            out += lit_len;
            in  += lit_len + lz_match.len;
          }
          memcpy(out, in, block->size - (in-buf));
          out += block->size - (in-buf);

          // Write compressed block literals to compressed file
          checked_file_write (fout, outbuf, out-outbuf);

if (FUTURE_LZ)
        {
          block->header[2] = stat_size;
          checked_file_write (fout, block->header, header_size);
        }