zpaq updates

[fcorbelli]

can you do multiple launches?, you should reset the system file cache before first run

This is not so useful, for me, because on clients I use only NVMe/SSD disks.
Can you please send me your EXE, or try mine?
http://www.francocorbelli.it/zpaqlist.exe

zpaqlist l "h:\zarc\copia_zarc.zpaq" -out z:\default.txt
zpaqlist l "h:\zarc\copia_zarc.zpaq"  -all -out z:\all.txt
zpaqlist l "h:\zarc\copia_zarc.zpaq"  -until 10 -out z:\10.txt

I attach the source, if you want to compile yourself.

The output (-all) of 715 is sorted by version, then by file.
Mine is sorted by file, then by version (aka: like a Time Machine).

To reduce time to write (and to read) from disk I "deduplicate" the filenames to just "?"
(invalid filename).

Write, and read, a 600MB (typically 715 list output of complex zpaq) file on magnetic disks takes time.
Shrinking to 170MB (my test bed) is faster, but not real quick.

---
Result: my PAKKA Windows GUI is much faster than anything else I have founded.
Of course... only half dozen competitors
But it doesn't satisfy me anyway

[SpyFX]

libzpaq.h.txt ? LIBZPAQ Version 7.00 header - Dec. 15, 2014.

libzpaq.cpp ? LIBZPAQ Version 6.52 implementation - May 9, 2014.

last version =>
​
libzpaq.h - LIBZPAQ Version 7.12 header - Apr. 19, 2016.
libzpaq.cpp - LIBZPAQ Version 7.15 implementation - Aug. 17, 2016.

[fcorbelli]

libzpaq.h.txt ? LIBZPAQ Version 7.00 header - Dec. 15, 2014.

This forum does not allow to attach .h.
Just rename libzpaq.h.txt to libzpaq.h


libzpaq.cpp ? LIBZPAQ Version 6.52 implementation - May 9, 2014.

last version =>
​
libzpaq.h - LIBZPAQ Version 7.12 header - Apr. 19, 2016.
libzpaq.cpp - LIBZPAQ Version 7.15 implementation - Aug. 17, 2016.

My zpaqlist is NOT based on 7.15 source (zpaq.cpp, libzpaq.cpp, libzpaq.h)

BUT

on 6.60 (with older and coordinated libzpaq)

Why?
https://encode.su/threads/456-zpaq-u...ll=1#post66588
Because enumerate files in different ways vs 7.15 (useless for a GUI).
I removed (almost) all the compression portion, leaving the listing one (rewritten the mylist () function) and some remnants of various commands that no longer exist.

Unfortunately the sources of the various versions of ZPAQ, and libzpaq, have very subtle incompatibilities, sometimes really difficult to find even for me (different default parameters, same name, different functionality etc).

So it is not immediate, at least for me, to bring a 6.60 source to work with the latest libzpaq and even less "merge" with the 7.15 source.

So it took me much less time (and effort) to keep the sources (6.60-franz22 and 7.15-franz42) distinct.

The first is zpaqlist.exe, the second zpaqfranz.exe

[SpyFX]

Z:\ZPAQ\backup>..\zpaqlist.exe l DISK_F_Y_????.zpaq -all -out 1.txt

29/11/2020 00:31:22 zpaqlist franz22

29/11/2020 00:31:22 :default

29/11/2020 00:32:08 OUTPUT...V 778, F 613125, 764.858.961.314 bytes 5931 blocks

29/11/2020 00:32:09 W 010% 00061313/00613125

29/11/2020 00:32:09 W 020% 00122626/00613125

29/11/2020 00:32:09 W 030% 00183939/00613125

29/11/2020 00:32:10 W 040% 00245252/00613125

29/11/2020 00:32:10 W 050% 00306565/00613125

29/11/2020 00:32:10 W 060% 00367878/00613125

29/11/2020 00:32:10 W 070% 00429191/00613125

29/11/2020 00:32:11 W 080% 00490504/00613125

29/11/2020 00:32:11 W 090% 00551817/00613125

29/11/2020 00:32:12 T=49.672s (all OK)​



Z:\ZPAQ\backup>..\zpaq715 l DISK_F_Y_????.zpaq -all > zpaq715.txt

38.484 seconds (all OK)

[fcorbelli]

Z:\ZPAQ\backup>..\zpaqlist.exe l DISK_F_Y_????.zpaq -all -out 1.txt

29/11/2020 00:31:22 zpaqlist franz22

29/11/2020 00:31:22 :default

29/11/2020 00:32:08 OUTPUT...V 778, F 613125, 764.858.961.314 bytes 5931 blocks

29/11/2020 00:32:09 W 010% 00061313/00613125

29/11/2020 00:32:09 W 020% 00122626/00613125

29/11/2020 00:32:09 W 030% 00183939/00613125

29/11/2020 00:32:10 W 040% 00245252/00613125

29/11/2020 00:32:10 W 050% 00306565/00613125

29/11/2020 00:32:10 W 060% 00367878/00613125

29/11/2020 00:32:10 W 070% 00429191/00613125

29/11/2020 00:32:11 W 080% 00490504/00613125

29/11/2020 00:32:11 W 090% 00551817/00613125

29/11/2020 00:32:12 T=49.672s (all OK)​



Z:\ZPAQ\backup>..\zpaq715 l DISK_F_Y_????.zpaq -all > zpaq715.txt

38.484 seconds (all OK)

Second round (cache loaded)?
Thank you

[SpyFX]

Second round (cache loaded)?
Thank you

no, second it's third test...

[fcorbelli]

no, second it's third test...

C:\zpaqfranz\660>zpaqlist l h:\zarc\copia_zarc.zpaq -all -out z:\1.txt

28/11/2020 23:09:43 zpaqlist franz22

28/11/2020 23:09:43 :default

28/11/2020 23:09:52 OUTPUT...V 1061, F 65628, 5.732.789.993 bytes 554 blocks

28/11/2020 23:09:53 W 010% 00006563/00065628

28/11/2020 23:09:53 W 020% 00013126/00065628

28/11/2020 23:09:54 W 030% 00019689/00065628

28/11/2020 23:09:54 W 040% 00026252/00065628

28/11/2020 23:09:55 W 050% 00032815/00065628

28/11/2020 23:09:55 W 060% 00039378/00065628

28/11/2020 23:09:56 W 070% 00045941/00065628

28/11/2020 23:09:56 W 080% 00052504/00065628

28/11/2020 23:09:58 W 090% 00059067/00065628

28/11/2020 23:09:59 T=16.062s (all OK)



C:\zpaqfranz\660>c:\nz\zpaq64 l h:\zarc\copia_zarc.zpaq -all >z:\2.txt

78.391 seconds (all OK)

[SpyFX]

all blocks from hdd =>

Z:\ZPAQ\backup>..\zpaqlist.exe l DISK_F_Y_????.zpaq -all -out 1.txt

29/11/2020 01:13:24 zpaqlist franz22

29/11/2020 01:13:24 :default

29/11/2020 01:14:32 OUTPUT...V 778, F 613125, 764.858.961.314 bytes 5931 blocks

29/11/2020 01:14:33 W 010% 00061313/00613125

29/11/2020 01:14:33 W 020% 00122626/00613125

29/11/2020 01:14:33 W 030% 00183939/00613125

29/11/2020 01:14:33 W 040% 00245252/00613125

29/11/2020 01:14:34 W 050% 00306565/00613125

29/11/2020 01:14:34 W 060% 00367878/00613125

29/11/2020 01:14:34 W 070% 00429191/00613125

29/11/2020 01:14:34 W 080% 00490504/00613125

29/11/2020 01:14:35 W 090% 00551817/00613125

29/11/2020 01:14:35 T=71.765s (all OK)

p/s 16sec looks like mystic...

[fcorbelli]

all blocks from hdd =>


p/s 16sec looks like mystic...

Those are normal performances on my pc

Tomorrow I will make a more verbose timing log.
Just to see the size of the output file.
good night

[fcorbelli]

I would need help finding, without having to study the source too long, where the fragments are embedded in the blocks, before being compressed.
Referring to 7.15 here

      libzpaq::compressBlock(&cj.in, &cj.out, cj.method.c_str(),

          cj.filename.c_str(), cj.comment=="" ? 0 : cj.comment.c_str());

the blocks are compressed, which should be comprised of a certain number of fragments.
But ... where are the fragments?
I have to calculate the CRC32 before they are placed in a block.

    crc=crc32_16bytes (cj.in.c_str(),cj.in.size());

I hope it's clear

[SpyFX]

    // Read fragments

    int64_t fsize=0;  // file size after dedupe

    for (unsigned fj=0; true; ++fj) {

      int64_t sz=0;  // fragment size;

      unsigned hits=0;  // correct prediction count

      int c=EOF;  // current byte

      unsigned htptr=0;  // fragment index

      char sha1result[20]={0};  // fragment hash

      unsigned char o1[256]={0};  // order 1 context -> predicted byte

      if (fi<vf.size()) {

        int c1=0;  // previous byte

        unsigned h=0;  // rolling hash for finding fragment boundaries

        libzpaq::SHA1 sha1;

        assert(in!=FPNULL);

        while (true) {

          if (bufptr>=buflen) bufptr=0, buflen=fread(buf, 1, BUFSIZE, in);

          if (bufptr>=buflen) c=EOF;

          else c=(unsigned char)buf[bufptr++];

          if (c!=EOF) {

            if (c==o1[c1]) h=(h+c+1)*314159265u, ++hits;

            else h=(h+c+1)*271828182u;

            o1[c1]=c;

            c1=c;

            sha1.put(c);

            fragbuf[sz++]=c;

          }

          if (c==EOF

              || sz>=MAX_FRAGMENT

              || (fragment<=22 && h<(1u<<(22-fragment)) && sz>=MIN_FRAGMENT))

            break;

        }

        assert(sz<=MAX_FRAGMENT);

        total_done+=sz;





        // Look for matching fragment

        assert(uint64_t(sz)==sha1.usize());

        memcpy(sha1result, sha1.result(), 20);

        htptr=htinv.find(sha1result);

      }  // end if fi<vf.size()



// >>>> this is the point at which the fragment is ready <<<< 



      if (htptr==0) {  // not matched or last block

.........

fragbuf[] and size=sz

[fcorbelli]

Easier navigation between different versions. Timestamps for all.

Hello,
I have trouble navigating by using "version" indexes only. Would it be possible to assign short version comments or names?


Then there are the dates. I am not interested in archiving dates. I may re-archive the files later and the archiving dates would not represent the historical dates of the files. If I want to set version dates when the files were backed up via "-until", then ZPAQ shoots at me with "zpaq error: cannot truncate with an index". I obviously don't want to truncate anything, much less so while creating new archive. I don't want to mess with system time for each version addition either. I only want to set the timestamp. I also want to use multi-part archives with an index. So many great features. But the "-until" switch looks overused for many different things.

docs regarding current state:


ideal example:

zpaq add BigArchive???.zpaq c:\Largefileset\* -m4 -t4 -f -index  BigArchive000.zpaq -until 2003-01-02 -comment version-1.0

Thanks.

OK, with "only" 2.5 years I suggest my solution (/workaround)

Adding a "fake" comment file, just after all insert.
Marked as $data (do not show in "normal" list) and with a 0 date (deleted file),
so it will not be extracted.
Snipped

// Append compressed index to archive

  int added=0;  // count

  for (DTMap::iterator p=edt.begin();; ++p) 

  {

    if (p!=edt.end()) 

    {

 (...)        string filename=rename(p->first);

        

    }

    else

    {

        if (versioncomment.length()>0)

        {

            string fakefile=versioncomment+":$DATA"; //hidden windows file

            puti(is, 0, 8); // this is the "date". 0 is good [DELETED], but do not pass paranoid compliance test.

                                // when listing take as version comment

            is.write(fakefile.c_str(), strlen(fakefile.c_str()));

            is.put(0);

            puti(is, 0, 4);  // no attributes

            puti(is, 0, 4);  // list of frag pointers

         }

    }

  }

Setted by a trivial "-comment" switch, and a "smarter" list() function.
If you find a 0-date 0-byte long :$DATA file, that's a version comment.

Should be 715 compatible.

Any ideas to improve?

[SpyFX]

if (versioncomment.length()>0)

        {

            string fakefile=versioncomment+":$DATA"; //hidden windows file

            puti(is, 0, 8); // this is the "date". 0 is good [DELETED], but do not pass paranoid compliance test.

                                // when listing take as version comment

            is.write(fakefile.c_str(), strlen(fakefile.c_str()));

            is.put(0);

//puti(is, 0, 4);  // no attributes

//puti(is, 0, 4);  // list of frag pointers

         }

if the file name is marked as date = 0 then the record should contain only the file name (zpaq code)

[fcorbelli]

if (versioncomment.length()>0)

        {

            string fakefile=versioncomment+":$DATA"; //hidden windows file

            puti(is, 0, 8); // this is the "date". 0 is good [DELETED], but do not pass paranoid compliance test.

                                // when listing take as version comment

            is.write(fakefile.c_str(), strlen(fakefile.c_str()));

            is.put(0);

//puti(is, 0, 4);  // no attributes

//puti(is, 0, 4);  // list of frag pointers

         }

if the file name is marked as date = 0 then the record should contain only the file name (zpaq code)

I think you are right and, in this case, I have to set date=0, otherwise the fake file will be extracted

In summary, I added a switch comment, both to add and to list, possibly with -all

C:\zpaqfranz>zpaqfranz l z:\1.zpaq -comment

zpaqfranz v7.15-franz44 journaling archiver, compiled Dec 11 2020

franz:use comment

z:/1.zpaq: 5 versions, 2008 files, 9485 fragments, 259.942006 MB



Version comments enumerator

00000001 <<prima_versione>>

00000002 <<seconda_versione>>

00000003 <<terza_versione>>

00000005 <<quinta_versione>>



0.031 seconds (all OK)









C:\zpaqfranz>zpaqfranz l z:\1.zpaq -all -comment

zpaqfranz v7.15-franz44 journaling archiver, compiled Dec 11 2020

franz:use comment

z:/1.zpaq: 5 versions, 2008 files, 9485 fragments, 259.942006 MB



Version comments enumerator

00000001 <<prima_versione>>

00000002 <<seconda_versione>>

00000003 <<terza_versione>>

00000005 <<quinta_versione>>



------

00000001 2020-12-11 08:03:55  +00000055 -00000001 ->            8.144.142 <<prima_versione>>

00000002 2020-12-11 08:04:05  +00000241 -00000001 ->            4.026.567 <<seconda_versione>>

00000003 2020-12-11 08:04:20  +00001653 -00000001 ->          241.510.262 <<terza_versione>>

00000004 2020-12-11 08:28:28  +00000019 -00000000 ->               66.826

00000005 2020-12-11 08:28:55  +00000040 -00000001 ->            6.194.209 <<quinta_versione>>



0.047 seconds (all OK)

C:\zpaqfranz>zpaqfranz a z:\1.zpaq c:\1.txt -comment sesta_versione

The encode is like this

if (versioncomment.length()>0)

        {

            ///VCOMMENT 00000002 seconda_versione:$DATA

            char versioni8[9];

            sprintf(versioni8,"%08d",ver.size());

            versioni8[8]=0x0;

            string fakefile="VCOMMENT "+string(versioni8)+" "+versioncomment+":$DATA"; //hidden windows file

            puti(is, 0, 8); // this is the "date". 0 is good, but do not pass paranoid compliance test. damn

            is.write(fakefile.c_str(), strlen(fakefile.c_str()));

            is.put(0);

            //puti(is, 0, 4);  // no attributes

//puti(is, 0, 4);  // list of frag pointers

      

        }

The decode function is very rude

map<int, string> mappacommenti;

           

    ///VCOMMENT 00000002 seconda_versione:$DATA

    for (unsigned i=0;i<filelist.size();i++) 

    {

        DTMap::iterator p=filelist[i];

        if (strstr(p->first.c_str(), ":$DATA"))

        {

            string fakefile=p->first;

            myreplace(fakefile,":$DATA","");

            size_t found = fakefile.find("VCOMMENT "); 

            if (found != string::npos)

            {

                string numeroversione=fakefile.substr(found+9,8);

                int numver=stoi(numeroversione.c_str());

                string commento=fakefile.substr(found+9+8+1,65000);

                mappacommenti.insert(pair<int, string>(numver, commento));

            }

        }

    }

    

I need to work on "string escaping" of the comments (maybe even mime64), and thorough testing.
Finally, a search in extraction by comment.
Find the #version, set -until X, extract

[fcorbelli]

AARRGG
Nothing is easy with ZPAQ.
Internal functions write almost everywhere, therefore this

    int64_t sz=read_archive(archive.c_str());

    version=1;

    sz=read_archive(archive.c_str());

does NOT work

but this

    version=1;

    int64_t sz=read_archive(archive.c_str());

yes.

So I need to patch a lot of messy code

[SpyFX]

I need to work on "string escaping" of the comments (maybe even mime64), and thorough testing.

my yEncAlgoritm

I think this is the most optimal

public static IEnumerable<byte> pack(byte[] bufferIn, int offset, int count)

{

    for (int i = offset; i < (offset + count); i++)

    {

        byte converted = (byte)((bufferIn[i] + 42) % 256);





        if (converted <= 0x01)

        {

            yield return 0x01;

            converted = (byte)((converted + 64) % 256);

        }





        yield return converted;

    }

}

bool bEscape = false;

public IEnumerable<byte> unpack(bool bInit, byte[] bufferIn, int offset, int count)

{

    if (bInit) bEscape = false;





    for (int i = offset; i < (offset + count); i++)

    {

        byte ch = bufferIn[i];





        if (bufferIn[i] == 0x01 && !bEscape)

        {

            bEscape = true; continue;

        }





        if (bEscape)

        {

            ch -= (byte)64; bEscape = false;

        }





        if (ch > 0 && ch < 42)

        {

            ch += (byte)214;

        }

        else

        {

            ch -= (byte)42;

        }





        yield return ch;

    }

}

[fcorbelli]

AARRGG
Nothing is easy with ZPAQ.
Internal functions write almost everywhere, therefore this

    int64_t sz=read_archive(archive.c_str());

    version=1;

    sz=read_archive(archive.c_str());

does NOT work

but this

    version=1;

    int64_t sz=read_archive(archive.c_str());

yes.

So I need to patch a lot of messy code

... and the answer seems...

if (flagcomment)

        if (versioncomment.length()>0)

        {

            vector<DTMap::iterator> myfilelist;

            int versione=searchcomments(versioncomment,myfilelist);

            if (versione>0)

            {

                printf("Found version -until %d scanning again...\n",versione);

                version=versione;

    

                ver.clear();

                block.clear();

                dt.clear();

                ht.clear();

                ht.resize(1);  

                ver.resize(1); 

                dhsize=dcsize=0;

                sz=read_archive(archive.c_str());

            }

            else

            

            error("Cannot find version comment");

        }

Not very fast, but works...

zpaqfranz x z:\1.zpaq -to z:\ugo -comment xb



zpaqfranz v7.15-franz44 journaling archiver, compiled Dec 11 2020

franz:use comment

z:/1.zpaq: 4 versions, 1988 files, 9487 fragments, 259.911744 MB

Found version -until 4 scanning again...

z:/1.zpaq -until 4: 4 versions, 1988 files, 9487 fragments, 259.911744 MB

Extracting 659.312716 MB in 1822 files -threads 12

(...)

[SpyFX]

if possible, test my application on h:\zarc\copia_zarc.zpaq

this is a very crude application, many functions but only two in working order

zpaqBackup -archive=test.zpaq -rebuild
or
zpaqBackup -archive=test.zpaq -rebuild -s3

a new archive is created (test.rd.zpaq)

zpaqBackup -archive=test.zpaq -list -all > test.zpaq.lst
zpaqBackup -archive=test.rd.zpaq -list -all > test.rd.zpaq.lst

Z:\test>zpaqBackup.exe -archive=.\ar\DISK_F_Y_???? -rebuild -s3

Load archive time: 00:02.433

▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓

100,00% d/block    2,505,091 |  2443639:?? =>  2443639:?? |  97.5 % =>  97.5 % |   0s | S/1296236 04:32 / 00:00 

Total time: 04:35, 'DISK_F_Y_0001.rd.zpaq' (21244.379136 MB) / 100.00 %



Z:\test>zpaqBackup.exe -archive=.\ar\DISK_F_Y_????.zpaq -list -all > 1.txt

Load archive time: 00:02.576

Total time: 00.00:03.212



Z:\test>zpaqBackup.exe -archive=.\ar\DISK_F_Y_????.rd.zpaq -list -all > 1.txt

Load archive time: 00:00.754

Total time: 00.00:01.357

it's .net 5.0.0 app https://dotnet.microsoft.com/downloa...-x64-installer

[fcorbelli]

C:\zpaqBackupTest>zpaqBackup.exe -archive=r:\prova\copia_zarc.zpaq -rebuild >rebuild.txt



Load archive time: 00:06.491



C:\zpaqBackupTest>zpaqBackup.exe -archive=r:\prova\copia_zarc.zpaq -rebuild -s3 >rebuilds3.txt



Load archive time: 00:06.648

rebuild and rebuild -s3 make the same output

Rebuilding archive file 'copia_zarc.zpaq' (5845.336904 MB) - threads 12
d/block   16,734,585 | 16735715:?? => 16735715:?? | 100.0 % => 100.0 % |   0s | S/1
d/block   16,762,600 | 16763730:?? => 16763730:?? | 100.0 % => 100.0 % |   0s | S/274
d/block   16,765,215 | 16766345:?? => 16766345:?? | 100.0 % => 100.0 % |   0s | S/531
d/block    2,169,596 |  2169837:?? =>  2169837:?? | 100.0 % => 100.0 % |   0s | S/764
d/block   16,754,970 |  9890504:?? =>  9890504:?? |  59.0 % =>  59.0 % |   0s | S/792
d/block    4,519,363 |  1522549:?? =>  1522549:?? |  33.7 % =>  33.7 % |   0s | S/1072
d/block   16,568,157 | 16569275:?? => 16569275:?? | 100.0 % => 100.0 % |   0s | S/2807
d/block   16,731,677 | 16732807:?? => 16732807:?? | 100.0 % => 100.0 % |   0s | S/3038
d/block   16,763,440 | 16764570:?? => 16764570:?? | 100.0 % => 100.0 % |   0s | S/3311
(...)
d/block   16,645,854 | 16646976:?? => 16646976:?? | 100.0 % => 100.0 % |   0s | S/166721
d/block   16,282,576 | 16283678:?? => 16283678:?? | 100.0 % => 100.0 % |   0s | S/166954
d/block   15,040,207 | 15041233:?? => 15041233:?? | 100.0 % => 100.0 % |   0s | S/167187
d/block    5,274,670 |  2908616:?? =>  2908616:?? |  55.1 % =>  55.1 % |   0s | S/167394
d/block   16,760,897 | 16762027:?? => 16762027:?? | 100.0 % => 100.0 % |   0s | S/167452
d/block   16,771,086 | 16772216:?? => 16772216:?? | 100.0 % => 100.0 % |   0s | S/167719
d/block    3,900,971 |  3901316:?? =>  3901316:?? | 100.0 % => 100.0 % |   0s | S/167945
d/block   12,442,240 | 12443106:?? => 12443106:?? | 100.0 % => 100.0 % |   0s | S/168001

Total time: 00:46, 'copia_zarc.rd.zpaq' (5860.57728 MB) / 100.26 %

 Directory di R:\prova



11/12/2020  15:05    <DIR>          .

11/12/2020  15:05    <DIR>          ..

11/12/2020  15:06     5.860.577.280 copia_zarc.rd.zpaq

11/12/2020  15:05     5.860.577.280 copia_zarc.rd.zpaq_uno

11/12/2020  14:30     5.845.336.904 copia_zarc.zpaq

               3 File 17.566.491.464 byte

C:\zpaqBackupTest>zpaqBackup.exe r:\prova\copia_zarc.rd.zpaq -list -all >listall.txt
Unhandled exception. System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
   at zpaqBackup.zpaqBackup.EnumerateFiles(CmdArguments cmd, List`1 versions, UInt32 archiveCpnMax)+MoveNext()
   at zpaqBackup.zpaqBackup.loadArchiveFast(CmdArguments cmd) in D:\diskc\repos\ZPAQ\zpaqBackup\LoadArchive.cs:line 849
   at zpaqBackup.zpaqBackup.ListArchive(CmdArguments cmd) in D:\diskc\repos\ZPAQ\zpaqBackup\ListArchive.cs:line 14
   at zpaqBackup.zpaqBackup.Main(String[] args) in D:\diskc\repos\ZPAQ\zpaqBackup\zpaqBackup.cs:line 33

[SpyFX]

-rebuild w/o -s3 d/block list in console as progress

-rebuild with -s3 progress bar


​
​zpaqBackup.exe archive=r:\prova\copia_zarc.zpaq -list -all >listall.txt
zpaqBackup.exe archive=r:\prova\copia_zarc.rd.zpaq -list -all >listall.rd.txt

[fcorbelli]

\zpaqBackupTest>zpaqBackup.exe archive=r:\prova\copia_zarc.zpaq -list -all >listall.txt

Unhandled exception. System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')

   at zpaqBackup.zpaqBackup.EnumerateFiles(CmdArguments cmd, List`1 versions, UInt32 archiveCpnMax)+MoveNext()

   at zpaqBackup.zpaqBackup.loadArchiveFast(CmdArguments cmd) in D:\diskc\repos\ZPAQ\zpaqBackup\LoadArchive.cs:line 849

   at zpaqBackup.zpaqBackup.ListArchive(CmdArguments cmd) in D:\diskc\repos\ZPAQ\zpaqBackup\ListArchive.cs:line 14

   at zpaqBackup.zpaqBackup.Main(String[] args) in D:\diskc\repos\ZPAQ\zpaqBackup\zpaqBackup.cs:line 33

C:\zpaqBackupTest>zpaqBackup.exe archive=r:\prova\copia_zarc.rd.zpaq -list -all >listall.rd.txt

Unhandled exception. System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')

   at zpaqBackup.zpaqBackup.EnumerateFiles(CmdArguments cmd, List`1 versions, UInt32 archiveCpnMax)+MoveNext()

   at zpaqBackup.zpaqBackup.loadArchiveFast(CmdArguments cmd) in D:\diskc\repos\ZPAQ\zpaqBackup\LoadArchive.cs:line 849

   at zpaqBackup.zpaqBackup.ListArchive(CmdArguments cmd) in D:\diskc\repos\ZPAQ\zpaqBackup\ListArchive.cs:line 14

   at zpaqBackup.zpaqBackup.Main(String[] args) in D:\diskc\repos\ZPAQ\zpaqBackup\zpaqBackup.cs:line 33

[SpyFX]

r:\prova\copia_zarc.zpaq exist?

update zpaqBackup.dll I added a check that the archive file exists

[fcorbelli]

C:\zpaqBackupTest>dir r:\prova\copia_zarc.zpaq
 Il volume nell'unità R non ha etichetta.
 Numero di serie del volume: 5655-23AB

 Directory di r:\prova

11/12/2020  14:30     5.845.336.904 copia_zarc.zpaq
               1 File  5.845.336.904 byte
               0 Directory  35.959.898.112 byte disponibili

C:\zpaqBackupTest>zpaqBackup.exe archive=r:\prova\copia_zarc.zpaq -list -all >listall.txt
.zpaq does not exist

[SpyFX]

C:\zpaqBackupTest>dir r:\prova\copia_zarc.zpaq
 Il volume nell'unità R non ha etichetta.
 Numero di serie del volume: 5655-23AB

 Directory di r:\prova

11/12/2020  14:30     5.845.336.904 copia_zarc.zpaq
               1 File  5.845.336.904 byte
               0 Directory  35.959.898.112 byte disponibili

C:\zpaqBackupTest>zpaqBackup.exe archive=r:\prova\copia_zarc.zpaq -list -all >listall.txt
.zpaq does not exist

sorry, forgot to put a dash in front of the parameter

​zpaqBackup.exe -archive=r:\prova\copia_zarc.zpaq -list -all >listall.txt
zpaqBackup.exe -archive=r:\prova\copia_zarc.rd.zpaq -list -all >listall.rd.txt

[fcorbelli]

First test

R:\prova>c:\zpaqfranz\660\zpaqlist.exe l copia_zarc.zpaq -all -out 2.txt
11/12/2020 19:37:37 zpaqlist franz23
11/12/2020 19:37:37 Multipart scan 0.000000 s chunks 1
11/12/2020 19:37:37 :all
11/12/2020 19:37:46 OUTPUT...V 1077, F 65634, 5.845.336.904 bytes 554 blocks Otime 0.000 s
11/12/2020 19:37:47 W 010% 00006493/00064921
11/12/2020 19:37:47 W 020% 00012986/00064921
11/12/2020 19:37:47 W 030% 00019479/00064921
11/12/2020 19:37:48 W 040% 00025972/00064921
11/12/2020 19:37:48 W 050% 00032465/00064921
11/12/2020 19:37:49 W 060% 00038958/00064921
11/12/2020 19:37:49 W 070% 00045451/00064921
11/12/2020 19:37:49 W 080% 00051944/00064921
11/12/2020 19:37:50 W 090% 00058437/00064921
11/12/2020 19:37:50 Output 182.744.963 bytes / 27.525.696 lines in 4.250 s
11/12/2020 19:37:51 T=14.391s (all OK)

R:\prova>c:zpaqBackup.exe -archive=copia_zarc.rd.zpaq -list -all >1.txt

Load archive time: 00:05.015
Total time: 00.00:15.912

R:\prova>c:\zpaqfranz\660\zpaqlist.exe l copia_zarc.rd.zpaq
11/12/2020 19:38:33 zpaqlist franz23
11/12/2020 19:38:33 Multipart scan 0.000000 s chunks 1
11/12/2020 19:38:33 :default
Bad JIDAC header size: 3997

[SpyFX]

First test

R:\prova>c:\zpaqfranz\660\zpaqlist.exe l copia_zarc.rd.zpaq
11/12/2020 19:38:33 zpaqlist franz23
11/12/2020 19:38:33 Multipart scan 0.000000 s chunks 1
11/12/2020 19:38:33 :default
Bad JIDAC header size: 3997

in .rd c/block usize=3997 bytes (4096 with header + sha1), in zpaq usize=8 bytes.. in source code zpaq715 dump error if usize < 8 only

fix for zpaqlist.cpp

            if (os.size() < 8) {

              printf( "Bad JIDAC header size: %d\n", list_size(os));

              isbreak=true;

              if (*errors) ++*errors;

            }

also in .rd there is a new type of block z/block, it is desirable to skip such blocks so that there are no messages about unknown blocks
​

[SpyFX]

my test with archive 508 files

[fcorbelli]

OK, this is my (almost) complete zpaqfranz
(not checked compiling on *nix, only Windows)

Need some rework on VSS (stripped down C++ streams, too big after linking).
http://www.francocorbelli.it/zpaqfranz.exe
http://www.francocorbelli.it/zpaqfranz32.exe

Not tested at all, but with (almost) everything I need

zpaqfranz v7.15-franz45 journaling archiver, compiled Dec 13 2020



Differences from ZPAQ 7.15

Changed behaviors:

0)  Output is somewhat different (-pakka for alternative)

1)  During add() zpaqfranz stores by default the CRC-32 of the files

    This can disabled by -crc32 switch

2)  Add() using -checksum will store SHA1 hash for every file,

    doing a CRC-32 check too

3)  By default every .XLS file is forcibily added (check of datetime

    is not reliable for ancient XLS to detect changes). Disabled by -xls

New functions:

4)  Using -comment something is possible to add ASCII text to the versions

    in add() and list(). WARNING: NO duplicates check is done

5)  In list() using -find pippo filter like |grep -i find

    In list() -comment / -comment pippo / -comment -all

6)  New command t (test) for archive test. -force for filesystem post-check

    -verbose

7)  New command p (paranoid test). Need LOTS of RAM and painfully slow. Check

    almost everything. -force -verbose

8)  New command s (size). Get the cumulative size of one or more directory.

    Useful on *nix, where it is not very quick to get this infos. Skip .zfs

    On *nix return (a kind of) free disk space on filesystem(s)

9)  New command sha1 (hashing). Calculate hash of something (default SHA1)

    -sha256, -crc32, -crc32c (HW SSE if possible), xxhash

10) New command dir. Something similar to dir command (for *nix)

    Switches /s /a /os /od. Show cumulative size in the last line!

11) New command help. -he show some examples, -diff differences from 7.15

12) -noeta. Do not show ETA (for batch file)

13) -pakka. Alternative output (for ZPAQ's GUI PAKKA)

14) -verbose. Show more info on files

15) -zfs. Skip ZFS's snapshots

16) -noqnap. Skip special directories

17) -nowindows. Turn off metafiles and system volume information

18) -always files. Always add some file

19) -nosort. Do not sort before adding files

20) -nopath. Extract in local path

21) -vss. On Windows (if running with admin power) take a snapshot (VSS) of C:

    drive before add. Useful for backup entire user profiles

22) -find something. Find text in full filename (ex list)

23) -replace something. Replace a -find text (manipulate output)

Essentially, the CRC-32 of each fragment is calculated (not cleverly, actually), and then recombined to form the CRC-32 of the whole files. They are then stored in my extension (backwards compatible, I hope) of the attributes.
There is no noticeable slowdown (lots of recalc running, but... it's a feature, not a bug!)

Using -checksum it is possible to force storage of SHA1 codes and to recalculate CRC-32s from files to disk. In this way, in practice, a check is obtained that the "whole" CRC-32 is consistent.
ZPAQ in its test does not (tests every single fragment with SHA1).

The -comment option adds (very crude) ASCII text to versions, and they can be used for the extraction phase, via a fake file I hope compatible with 7.15.

The dir command is a LONG waited dir clone (I use almost always Unix servers).

Any comments and tests are welcome.

Once the functions are "freezed", I will move on debugging and hardening.

[danieldeak70]

Might be stupid question but is there a way to turn off to not store the full path? Cause if i extract something it literally creates the folders for the path. Like: Creates a C folder inside that a Users floder and etc etc...

[fcorbelli]

Might be stupid question but is there a way to turn off to not store the full path? Cause if i extract something it literally creates the folders for the path. Like: Creates a C folder inside that a Users floder and etc etc...

-nopath