Page 88 of 89 FirstFirst ... 387886878889 LastLast
Results 2,611 to 2,640 of 2657

Thread: zpaq updates

  1. #2611
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts
    Quote Originally Posted by SpyFX View Post
    can you do multiple launches?, you should reset the system file cache before first run
    This is not so useful, for me, because on clients I use only NVMe/SSD disks.
    Can you please send me your EXE, or try mine?
    http://www.francocorbelli.it/zpaqlist.exe
    Code:
    zpaqlist l "h:\zarc\copia_zarc.zpaq" -out z:\default.txt
    zpaqlist l "h:\zarc\copia_zarc.zpaq"  -all -out z:\all.txt
    zpaqlist l "h:\zarc\copia_zarc.zpaq"  -until 10 -out z:\10.txt
    I attach the source, if you want to compile yourself.

    The output (-all) of 715 is sorted by version, then by file.
    Mine is sorted by file, then by version (aka: like a Time Machine).

    To reduce time to write (and to read) from disk I "deduplicate" the filenames to just "?"
    (invalid filename).

    Write, and read, a 600MB (typically 715 list output of complex zpaq) file on magnetic disks takes time.
    Shrinking to 170MB (my test bed) is faster, but not real quick.

    ---
    Result: my PAKKA Windows GUI is much faster than anything else I have founded.
    Of course... only half dozen competitors
    But it doesn't satisfy me anyway
    Attached Files Attached Files

  2. #2612
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    42
    Thanks
    4
    Thanked 7 Times in 6 Posts
    libzpaq.h.txt ? LIBZPAQ Version 7.00 header - Dec. 15, 2014.

    libzpaq.cpp ? LIBZPAQ Version 6.52 implementation - May 9, 2014.

    last version =>

    libzpaq.h - LIBZPAQ Version 7.12 header - Apr. 19, 2016.
    libzpaq.cpp - LIBZPAQ Version 7.15 implementation - Aug. 17, 2016.

  3. #2613
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts
    Quote Originally Posted by SpyFX View Post
    libzpaq.h.txt ? LIBZPAQ Version 7.00 header - Dec. 15, 2014.
    This forum does not allow to attach .h.
    Just rename libzpaq.h.txt to libzpaq.h


    libzpaq.cpp ? LIBZPAQ Version 6.52 implementation - May 9, 2014.
    last version =>

    libzpaq.h - LIBZPAQ Version 7.12 header - Apr. 19, 2016.
    libzpaq.cpp - LIBZPAQ Version 7.15 implementation - Aug. 17, 2016.
    My zpaqlist is NOT based on 7.15 source (zpaq.cpp, libzpaq.cpp, libzpaq.h)

    BUT

    on 6.60 (with older and coordinated libzpaq)

    Why?
    https://encode.su/threads/456-zpaq-u...ll=1#post66588
    Because enumerate files in different ways vs 7.15 (useless for a GUI).
    I removed (almost) all the compression portion, leaving the listing one (rewritten the mylist () function) and some remnants of various commands that no longer exist.

    Unfortunately the sources of the various versions of ZPAQ, and libzpaq, have very subtle incompatibilities, sometimes really difficult to find even for me (different default parameters, same name, different functionality etc).

    So it is not immediate, at least for me, to bring a 6.60 source to work with the latest libzpaq and even less "merge" with the 7.15 source.

    So it took me much less time (and effort) to keep the sources (6.60-franz22 and 7.15-franz42) distinct.

    The first is zpaqlist.exe, the second zpaqfranz.exe

  4. #2614
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    42
    Thanks
    4
    Thanked 7 Times in 6 Posts
    Z:\ZPAQ\backup>..\zpaqlist.exe l DISK_F_Y_????.zpaq -all -out 1.txt
    29/11/2020 00:31:22 zpaqlist franz22
    29/11/2020 00:31:22 :default
    29/11/2020 00:32:08 OUTPUT...V 778, F 613125, 764.858.961.314 bytes 5931 blocks
    29/11/2020 00:32:09 W 010% 00061313/00613125
    29/11/2020 00:32:09 W 020% 00122626/00613125
    29/11/2020 00:32:09 W 030% 00183939/00613125
    29/11/2020 00:32:10 W 040% 00245252/00613125
    29/11/2020 00:32:10 W 050% 00306565/00613125
    29/11/2020 00:32:10 W 060% 00367878/00613125
    29/11/2020 00:32:10 W 070% 00429191/00613125
    29/11/2020 00:32:11 W 080% 00490504/00613125
    29/11/2020 00:32:11 W 090% 00551817/00613125
    29/11/2020 00:32:12 T=49.672s (all OK)​

    Z:\ZPAQ\backup>..\zpaq715 l DISK_F_Y_????.zpaq -all > zpaq715.txt
    38.484 seconds (all OK)

  5. #2615
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts
    Quote Originally Posted by SpyFX View Post
    Z:\ZPAQ\backup>..\zpaqlist.exe l DISK_F_Y_????.zpaq -all -out 1.txt
    29/11/2020 00:31:22 zpaqlist franz22
    29/11/2020 00:31:22 :default
    29/11/2020 00:32:08 OUTPUT...V 778, F 613125, 764.858.961.314 bytes 5931 blocks
    29/11/2020 00:32:09 W 010% 00061313/00613125
    29/11/2020 00:32:09 W 020% 00122626/00613125
    29/11/2020 00:32:09 W 030% 00183939/00613125
    29/11/2020 00:32:10 W 040% 00245252/00613125
    29/11/2020 00:32:10 W 050% 00306565/00613125
    29/11/2020 00:32:10 W 060% 00367878/00613125
    29/11/2020 00:32:10 W 070% 00429191/00613125
    29/11/2020 00:32:11 W 080% 00490504/00613125
    29/11/2020 00:32:11 W 090% 00551817/00613125
    29/11/2020 00:32:12 T=49.672s (all OK)​

    Z:\ZPAQ\backup>..\zpaq715 l DISK_F_Y_????.zpaq -all > zpaq715.txt
    38.484 seconds (all OK)
    Second round (cache loaded)?
    Thank you

  6. #2616
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    42
    Thanks
    4
    Thanked 7 Times in 6 Posts
    Quote Originally Posted by fcorbelli View Post
    Second round (cache loaded)?
    Thank you
    no, second it's third test...

  7. #2617
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts
    Quote Originally Posted by SpyFX View Post
    no, second it's third test...
    C:\zpaqfranz\660>zpaqlist l h:\zarc\copia_zarc.zpaq -all -out z:\1.txt
    28/11/2020 23:09:43 zpaqlist franz22
    28/11/2020 23:09:43 :default
    28/11/2020 23:09:52 OUTPUT...V 1061, F 65628, 5.732.789.993 bytes 554 blocks
    28/11/2020 23:09:53 W 010% 00006563/00065628
    28/11/2020 23:09:53 W 020% 00013126/00065628
    28/11/2020 23:09:54 W 030% 00019689/00065628
    28/11/2020 23:09:54 W 040% 00026252/00065628
    28/11/2020 23:09:55 W 050% 00032815/00065628
    28/11/2020 23:09:55 W 060% 00039378/00065628
    28/11/2020 23:09:56 W 070% 00045941/00065628
    28/11/2020 23:09:56 W 080% 00052504/00065628
    28/11/2020 23:09:58 W 090% 00059067/00065628
    28/11/2020 23:09:59 T=16.062s (all OK)

    C:\zpaqfranz\660>c:\nz\zpaq64 l h:\zarc\copia_zarc.zpaq -all >z:\2.txt
    78.391 seconds (all OK)

  8. #2618
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    42
    Thanks
    4
    Thanked 7 Times in 6 Posts
    all blocks from hdd =>

    Z:\ZPAQ\backup>..\zpaqlist.exe l DISK_F_Y_????.zpaq -all -out 1.txt
    29/11/2020 01:13:24 zpaqlist franz22
    29/11/2020 01:13:24 :default
    29/11/2020 01:14:32 OUTPUT...V 778, F 613125, 764.858.961.314 bytes 5931 blocks
    29/11/2020 01:14:33 W 010% 00061313/00613125
    29/11/2020 01:14:33 W 020% 00122626/00613125
    29/11/2020 01:14:33 W 030% 00183939/00613125
    29/11/2020 01:14:33 W 040% 00245252/00613125
    29/11/2020 01:14:34 W 050% 00306565/00613125
    29/11/2020 01:14:34 W 060% 00367878/00613125
    29/11/2020 01:14:34 W 070% 00429191/00613125
    29/11/2020 01:14:34 W 080% 00490504/00613125
    29/11/2020 01:14:35 W 090% 00551817/00613125
    29/11/2020 01:14:35 T=71.765s (all OK)


    p/s 16sec looks like mystic...

  9. #2619
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts
    Quote Originally Posted by SpyFX View Post
    all blocks from hdd =>


    p/s 16sec looks like mystic...
    Those are normal performances on my pc

    Tomorrow I will make a more verbose timing log.
    Just to see the size of the output file.
    good night

  10. #2620
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts
    I would need help finding, without having to study the source too long, where the fragments are embedded in the blocks, before being compressed.
    Referring to 7.15 here

    libzpaq::compressBlock(&cj.in, &cj.out, cj.method.c_str(),
    cj.filename.c_str(), cj.comment=="" ? 0 : cj.comment.c_str());


    the blocks are compressed, which should be comprised of a certain number of fragments.
    But ... where are the fragments?
    I have to calculate the CRC32 before they are placed in a block.
        crc=crc32_16bytes (cj.in.c_str(),cj.in.size());


    I hope it's clear

  11. #2621
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    42
    Thanks
    4
    Thanked 7 Times in 6 Posts

    // Read fragments
    int64_t fsize=0; // file size after dedupe
    for (unsigned fj=0; true; ++fj) {
    int64_t sz=0; // fragment size;
    unsigned hits=0; // correct prediction count
    int c=EOF; // current byte
    unsigned htptr=0; // fragment index
    char sha1result[20]={0}; // fragment hash
    unsigned char o1[256]={0}; // order 1 context -> predicted byte
    if (fi<vf.size()) {
    int c1=0; // previous byte
    unsigned h=0; // rolling hash for finding fragment boundaries
    libzpaq::SHA1 sha1;
    assert(in!=FPNULL);
    while (true) {
    if (bufptr>=buflen) bufptr=0, buflen=fread(buf, 1, BUFSIZE, in);
    if (bufptr>=buflen) c=EOF;
    else c=(unsigned char)buf[bufptr++];
    if (c!=EOF) {
    if (c==o1[c1]) h=(h+c+1)*314159265u, ++hits;
    else h=(h+c+1)*271828182u;
    o1[c1]=c;
    c1=c;
    sha1.put(c);
    fragbuf[sz++]=c;
    }
    if (c==EOF
    || sz>=MAX_FRAGMENT
    || (fragment<=22 && h<(1u<<(22-fragment)) && sz>=MIN_FRAGMENT))
    break;
    }
    assert(sz<=MAX_FRAGMENT);
    total_done+=sz;


    // Look for matching fragment
    assert(uint64_t(sz)==sha1.usize());
    memcpy(sha1result, sha1.result(), 20);
    htptr=htinv.find(sha1result);
    } // end if fi<vf.size()

    // >>>> this is the point at which the fragment is ready <<<<

    if (htptr==0) { // not matched or last block
    .........



    fragbuf[] and size=sz

  12. Thanks:

    fcorbelli (10th December 2020)

  13. #2622
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts
    Quote Originally Posted by xpk View Post
    Easier navigation between different versions. Timestamps for all.

    Hello,
    I have trouble navigating by using "version" indexes only. Would it be possible to assign short version comments or names?


    Then there are the dates. I am not interested in archiving dates. I may re-archive the files later and the archiving dates would not represent the historical dates of the files. If I want to set version dates when the files were backed up via "-until", then ZPAQ shoots at me with "zpaq error: cannot truncate with an index". I obviously don't want to truncate anything, much less so while creating new archive. I don't want to mess with system time for each version addition either. I only want to set the timestamp. I also want to use multi-part archives with an index. So many great features. But the "-until" switch looks overused for many different things.

    docs regarding current state:


    ideal example:
    Code:
    zpaq add BigArchive???.zpaq c:\Largefileset\* -m4 -t4 -f -index  BigArchive000.zpaq -until 2003-01-02 -comment version-1.0
    Thanks.


    OK, with "only" 2.5 years I suggest my solution (/workaround)

    Adding a "fake" comment file, just after all insert.
    Marked as $data (do not show in "normal" list) and with a 0 date (deleted file),
    so it will not be extracted.
    Snipped


    // Append compressed index to archive
    int added=0; // count
    for (DTMap::iterator p=edt.begin();; ++p)
    {
    if (p!=edt.end())
    {
    (...) string filename=rename(p->first);

    }
    else
    {
    if (versioncomment.length()>0)
    {
    string fakefile=versioncomment+":$DATA"; //hidden windows file
    puti(is, 0, 8); // this is the "date". 0 is good [DELETED], but do not pass paranoid compliance test.
    // when listing take as version comment
    is.write(fakefile.c_str(), strlen(fakefile.c_str()));
    is.put(0);
    puti(is, 0, 4); // no attributes
    puti(is, 0, 4); // list of frag pointers
    }
    }
    }


    Setted by a trivial "-comment" switch, and a "smarter" list() function.
    If you find a 0-date 0-byte long :$DATA file, that's a version comment.

    Should be 715 compatible.

    Any ideas to improve?

  14. #2623
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    42
    Thanks
    4
    Thanked 7 Times in 6 Posts

    if (versioncomment.length()>0)
    {
    string fakefile=versioncomment+":$DATA"; //hidden windows file
    puti(is, 0, 8); // this is the "date". 0 is good [DELETED], but do not pass paranoid compliance test.
    // when listing take as version comment
    is.write(fakefile.c_str(), strlen(fakefile.c_str()));
    is.put(0);
    //puti(is, 0, 4); // no attributes
    //puti(is, 0, 4); // list of frag pointers
    }


    if the file name is marked as date = 0 then the record should contain only the file name (zpaq code)

  15. Thanks:

    fcorbelli (11th December 2020)

  16. #2624
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts
    Quote Originally Posted by SpyFX View Post

    if (versioncomment.length()>0)
    {
    string fakefile=versioncomment+":$DATA"; //hidden windows file
    puti(is, 0, 8); // this is the "date". 0 is good [DELETED], but do not pass paranoid compliance test.
    // when listing take as version comment
    is.write(fakefile.c_str(), strlen(fakefile.c_str()));
    is.put(0);
    //puti(is, 0, 4); // no attributes
    //puti(is, 0, 4); // list of frag pointers
    }


    if the file name is marked as date = 0 then the record should contain only the file name (zpaq code)
    I think you are right and, in this case, I have to set date=0, otherwise the fake file will be extracted

    In summary, I added a switch comment, both to add and to list, possibly with -all



    C:\zpaqfranz>zpaqfranz l z:\1.zpaq -comment
    zpaqfranz v7.15-franz44 journaling archiver, compiled Dec 11 2020
    franz:use comment
    z:/1.zpaq: 5 versions, 2008 files, 9485 fragments, 259.942006 MB

    Version comments enumerator
    00000001 <<prima_versione>>
    00000002 <<seconda_versione>>
    00000003 <<terza_versione>>
    00000005 <<quinta_versione>>

    0.031 seconds (all OK)




    C:\zpaqfranz>zpaqfranz l z:\1.zpaq -all -comment
    zpaqfranz v7.15-franz44 journaling archiver, compiled Dec 11 2020
    franz:use comment
    z:/1.zpaq: 5 versions, 2008 files, 9485 fragments, 259.942006 MB

    Version comments enumerator
    00000001 <<prima_versione>>
    00000002 <<seconda_versione>>
    00000003 <<terza_versione>>
    00000005 <<quinta_versione>>

    ------
    00000001 2020-12-11 08:03:55 +00000055 -00000001 -> 8.144.142 <<prima_versione>>
    00000002 2020-12-11 08:04:05 +00000241 -00000001 -> 4.026.567 <<seconda_versione>>
    00000003 2020-12-11 08:04:20 +00001653 -00000001 -> 241.510.262 <<terza_versione>>
    00000004 2020-12-11 08:28:28 +00000019 -00000000 -> 66.826
    00000005 2020-12-11 08:28:55 +00000040 -00000001 -> 6.194.209 <<quinta_versione>>

    0.047 seconds (all OK)



    Code:
    C:\zpaqfranz>zpaqfranz a z:\1.zpaq c:\1.txt -comment sesta_versione
    The encode is like this

    if (versioncomment.length()>0)
    {
    ///VCOMMENT 00000002 seconda_versione:$DATA
    char versioni8[9];
    sprintf(versioni8,"%08d",ver.size());
    versioni8[8]=0x0;
    string fakefile="VCOMMENT "+string(versioni8)+" "+versioncomment+":$DATA"; //hidden windows file
    puti(is, 0, 8); // this is the "date". 0 is good, but do not pass paranoid compliance test. damn
    is.write(fakefile.c_str(), strlen(fakefile.c_str()));
    is.put(0);
    //puti(is, 0, 4); // no attributes
    //puti(is, 0, 4); // list of frag pointers

    }

    The decode function is very rude

    map<int, string> mappacommenti;

    ///VCOMMENT 00000002 seconda_versione:$DATA
    for (unsigned i=0;i<filelist.size();i++)
    {
    DTMap::iterator p=filelist[i];
    if (strstr(p->first.c_str(), ":$DATA"))
    {
    string fakefile=p->first;
    myreplace(fakefile,":$DATA","");
    size_t found = fakefile.find("VCOMMENT ");
    if (found != string::npos)
    {
    string numeroversione=fakefile.substr(found+9,8);
    int numver=stoi(numeroversione.c_str());
    string commento=fakefile.substr(found+9+8+1,65000);
    mappacommenti.insert(pair<int, string>(numver, commento));
    }
    }
    }


    I need to work on "string escaping" of the comments (maybe even mime64), and thorough testing.
    Finally, a search in extraction by comment.
    Find the #version, set -until X, extract

  17. #2625
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts
    AARRGG
    Nothing is easy with ZPAQ.
    Internal functions write almost everywhere, therefore this


    int64_t sz=read_archive(archive.c_str());
    version=1;
    sz=read_archive(archive.c_str());

    does NOT work

    but this


    version=1;
    int64_t sz=read_archive(archive.c_str());

    yes.

    So I need to patch a lot of messy code

  18. #2626
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    42
    Thanks
    4
    Thanked 7 Times in 6 Posts

    Arrow

    Quote Originally Posted by fcorbelli View Post
    I need to work on "string escaping" of the comments (maybe even mime64), and thorough testing.
    my yEncAlgoritm

    I think this is the most optimal


    public static IEnumerable<byte> pack(byte[] bufferIn, int offset, int count)
    {
    for (int i = offset; i < (offset + count); i++)
    {
    byte converted = (byte)((bufferIn[i] + 42) % 256);


    if (converted <= 0x01)
    {
    yield return 0x01;
    converted = (byte)((converted + 64) % 256);
    }


    yield return converted;
    }
    }



    bool bEscape = false;
    public IEnumerable<byte> unpack(bool bInit, byte[] bufferIn, int offset, int count)
    {
    if (bInit) bEscape = false;


    for (int i = offset; i < (offset + count); i++)
    {
    byte ch = bufferIn[i];


    if (bufferIn[i] == 0x01 && !bEscape)
    {
    bEscape = true; continue;
    }


    if (bEscape)
    {
    ch -= (byte)64; bEscape = false;
    }


    if (ch > 0 && ch < 42)
    {
    ch += (byte)214;
    }
    else
    {
    ch -= (byte)42;
    }


    yield return ch;
    }
    }

  19. #2627
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts
    Quote Originally Posted by fcorbelli View Post
    AARRGG
    Nothing is easy with ZPAQ.
    Internal functions write almost everywhere, therefore this


    int64_t sz=read_archive(archive.c_str());
    version=1;
    sz=read_archive(archive.c_str());

    does NOT work

    but this


    version=1;
    int64_t sz=read_archive(archive.c_str());

    yes.

    So I need to patch a lot of messy code
    ... and the answer seems...

    if (flagcomment)
    if (versioncomment.length()>0)
    {
    vector<DTMap::iterator> myfilelist;
    int versione=searchcomments(versioncomment,myfilelist);
    if (versione>0)
    {
    printf("Found version -until %d scanning again...\n",versione);
    version=versione;

    ver.clear();
    block.clear();
    dt.clear();
    ht.clear();
    ht.resize(1);
    ver.resize(1);
    dhsize=dcsize=0;
    sz=read_archive(archive.c_str());
    }
    else

    error("Cannot find version comment");
    }


    Not very fast, but works...

    zpaqfranz x z:\1.zpaq -to z:\ugo -comment xb

    zpaqfranz v7.15-franz44 journaling archiver, compiled Dec 11 2020
    franz:use comment
    z:/1.zpaq: 4 versions, 1988 files, 9487 fragments, 259.911744 MB
    Found version -until 4 scanning again...
    z:/1.zpaq -until 4: 4 versions, 1988 files, 9487 fragments, 259.911744 MB
    Extracting 659.312716 MB in 1822 files -threads 12
    (...)


  20. #2628
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    42
    Thanks
    4
    Thanked 7 Times in 6 Posts
    if possible, test my application on h:\zarc\copia_zarc.zpaq

    this is a very crude application, many functions but only two in working order

    zpaqBackup -archive=test.zpaq -rebuild
    or
    zpaqBackup -archive=test.zpaq -rebuild -s3

    a new archive is created (test.rd.zpaq)

    zpaqBackup -archive=test.zpaq -list -all > test.zpaq.lst
    zpaqBackup -archive=test.rd.zpaq -list -all > test.rd.zpaq.lst


    Z:\test>zpaqBackup.exe -archive=.\ar\DISK_F_Y_???? -rebuild -s3
    Load archive time: 00:02.433
    ▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓
    100,00% d/block 2,505,091 | 2443639:?? => 2443639:?? | 97.5 % => 97.5 % | 0s | S/1296236 04:32 / 00:00
    Total time: 04:35, 'DISK_F_Y_0001.rd.zpaq' (21244.379136 MB) / 100.00 %

    Z:\test>zpaqBackup.exe -archive=.\ar\DISK_F_Y_????.zpaq -list -all > 1.txt
    Load archive time: 00:02.576
    Total time: 00.00:03.212

    Z:\test>zpaqBackup.exe -archive=.\ar\DISK_F_Y_????.rd.zpaq -list -all > 1.txt
    Load archive time: 00:00.754
    Total time: 00.00:01.357


    it's .net 5.0.0 app https://dotnet.microsoft.com/downloa...-x64-installer
    Attached Files Attached Files

  21. #2629
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts

    C:\zpaqBackupTest>zpaqBackup.exe -archive=r:\prova\copia_zarc.zpaq -rebuild >rebuild.txt

    Load archive time: 00:06.491

    C:\zpaqBackupTest>zpaqBackup.exe -archive=r:\prova\copia_zarc.zpaq -rebuild -s3 >rebuilds3.txt

    Load archive time: 00:06.648


    rebuild and rebuild -s3 make the same output

    Code:
    Rebuilding archive file 'copia_zarc.zpaq' (5845.336904 MB) - threads 12
    d/block   16,734,585 | 16735715:?? => 16735715:?? | 100.0 % => 100.0 % |   0s | S/1
    d/block   16,762,600 | 16763730:?? => 16763730:?? | 100.0 % => 100.0 % |   0s | S/274
    d/block   16,765,215 | 16766345:?? => 16766345:?? | 100.0 % => 100.0 % |   0s | S/531
    d/block    2,169,596 |  2169837:?? =>  2169837:?? | 100.0 % => 100.0 % |   0s | S/764
    d/block   16,754,970 |  9890504:?? =>  9890504:?? |  59.0 % =>  59.0 % |   0s | S/792
    d/block    4,519,363 |  1522549:?? =>  1522549:?? |  33.7 % =>  33.7 % |   0s | S/1072
    d/block   16,568,157 | 16569275:?? => 16569275:?? | 100.0 % => 100.0 % |   0s | S/2807
    d/block   16,731,677 | 16732807:?? => 16732807:?? | 100.0 % => 100.0 % |   0s | S/3038
    d/block   16,763,440 | 16764570:?? => 16764570:?? | 100.0 % => 100.0 % |   0s | S/3311
    (...)
    d/block   16,645,854 | 16646976:?? => 16646976:?? | 100.0 % => 100.0 % |   0s | S/166721
    d/block   16,282,576 | 16283678:?? => 16283678:?? | 100.0 % => 100.0 % |   0s | S/166954
    d/block   15,040,207 | 15041233:?? => 15041233:?? | 100.0 % => 100.0 % |   0s | S/167187
    d/block    5,274,670 |  2908616:?? =>  2908616:?? |  55.1 % =>  55.1 % |   0s | S/167394
    d/block   16,760,897 | 16762027:?? => 16762027:?? | 100.0 % => 100.0 % |   0s | S/167452
    d/block   16,771,086 | 16772216:?? => 16772216:?? | 100.0 % => 100.0 % |   0s | S/167719
    d/block    3,900,971 |  3901316:?? =>  3901316:?? | 100.0 % => 100.0 % |   0s | S/167945
    d/block   12,442,240 | 12443106:?? => 12443106:?? | 100.0 % => 100.0 % |   0s | S/168001
    
    Total time: 00:46, 'copia_zarc.rd.zpaq' (5860.57728 MB) / 100.26 %

    Directory di R:\prova

    11/12/2020 15:05 <DIR> .
    11/12/2020 15:05 <DIR> ..
    11/12/2020 15:06 5.860.577.280 copia_zarc.rd.zpaq
    11/12/2020 15:05 5.860.577.280 copia_zarc.rd.zpaq_uno
    11/12/2020 14:30 5.845.336.904 copia_zarc.zpaq
    3 File 17.566.491.464 byte



    Code:
    C:\zpaqBackupTest>zpaqBackup.exe r:\prova\copia_zarc.rd.zpaq -list -all >listall.txt
    Unhandled exception. System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
       at zpaqBackup.zpaqBackup.EnumerateFiles(CmdArguments cmd, List`1 versions, UInt32 archiveCpnMax)+MoveNext()
       at zpaqBackup.zpaqBackup.loadArchiveFast(CmdArguments cmd) in D:\diskc\repos\ZPAQ\zpaqBackup\LoadArchive.cs:line 849
       at zpaqBackup.zpaqBackup.ListArchive(CmdArguments cmd) in D:\diskc\repos\ZPAQ\zpaqBackup\ListArchive.cs:line 14
       at zpaqBackup.zpaqBackup.Main(String[] args) in D:\diskc\repos\ZPAQ\zpaqBackup\zpaqBackup.cs:line 33

  22. #2630
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    42
    Thanks
    4
    Thanked 7 Times in 6 Posts
    -rebuild w/o -s3 d/block list in console as progress

    -rebuild with -s3 progress bar

    Click image for larger version. 

Name:	PWXQD7_aTzyn7TXSh-ZXQw.png 
Views:	16 
Size:	7.5 KB 
ID:	8167

    ​zpaqBackup.exe archive=r:\prova\copia_zarc.zpaq -list -all >listall.txt

    zpaqBackup.exe archive=r:\prova\copia_zarc.rd.zpaq -list -all >listall.rd.txt
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	F5f1pX-7RzSReB2B4NJSWQ.png 
Views:	13 
Size:	14.5 KB 
ID:	8166  
    Last edited by SpyFX; 11th December 2020 at 18:36.

  23. #2631
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts

    \zpaqBackupTest>zpaqBackup.exe archive=r:\prova\copia_zarc.zpaq -list -all >listall.txt
    Unhandled exception. System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
    at zpaqBackup.zpaqBackup.EnumerateFiles(CmdArguments cmd, List`1 versions, UInt32 archiveCpnMax)+MoveNext()
    at zpaqBackup.zpaqBackup.loadArchiveFast(CmdArguments cmd) in D:\diskc\repos\ZPAQ\zpaqBackup\LoadArchive.cs:line 849
    at zpaqBackup.zpaqBackup.ListArchive(CmdArguments cmd) in D:\diskc\repos\ZPAQ\zpaqBackup\ListArchive.cs:line 14
    at zpaqBackup.zpaqBackup.Main(String[] args) in D:\diskc\repos\ZPAQ\zpaqBackup\zpaqBackup.cs:line 33



    C:\zpaqBackupTest>zpaqBackup.exe archive=r:\prova\copia_zarc.rd.zpaq -list -all >listall.rd.txt
    Unhandled exception. System.ArgumentOutOfRangeException: Index was out of range. Must be non-negative and less than the size of the collection. (Parameter 'index')
    at zpaqBackup.zpaqBackup.EnumerateFiles(CmdArguments cmd, List`1 versions, UInt32 archiveCpnMax)+MoveNext()
    at zpaqBackup.zpaqBackup.loadArchiveFast(CmdArguments cmd) in D:\diskc\repos\ZPAQ\zpaqBackup\LoadArchive.cs:line 849
    at zpaqBackup.zpaqBackup.ListArchive(CmdArguments cmd) in D:\diskc\repos\ZPAQ\zpaqBackup\ListArchive.cs:line 14
    at zpaqBackup.zpaqBackup.Main(String[] args) in D:\diskc\repos\ZPAQ\zpaqBackup\zpaqBackup.cs:line 33

  24. #2632
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    42
    Thanks
    4
    Thanked 7 Times in 6 Posts
    r:\prova\copia_zarc.zpaq exist?

    update zpaqBackup.dll I added a check that the archive file exists
    Attached Files Attached Files
    Last edited by SpyFX; 11th December 2020 at 20:57.

  25. #2633
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts
    Code:
    C:\zpaqBackupTest>dir r:\prova\copia_zarc.zpaq
     Il volume nell'unità R non ha etichetta.
     Numero di serie del volume: 5655-23AB
    
     Directory di r:\prova
    
    11/12/2020  14:30     5.845.336.904 copia_zarc.zpaq
                   1 File  5.845.336.904 byte
                   0 Directory  35.959.898.112 byte disponibili
    
    C:\zpaqBackupTest>zpaqBackup.exe archive=r:\prova\copia_zarc.zpaq -list -all >listall.txt
    .zpaq does not exist

  26. #2634
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    42
    Thanks
    4
    Thanked 7 Times in 6 Posts
    Quote Originally Posted by fcorbelli View Post
    Code:
    C:\zpaqBackupTest>dir r:\prova\copia_zarc.zpaq
     Il volume nell'unità R non ha etichetta.
     Numero di serie del volume: 5655-23AB
    
     Directory di r:\prova
    
    11/12/2020  14:30     5.845.336.904 copia_zarc.zpaq
                   1 File  5.845.336.904 byte
                   0 Directory  35.959.898.112 byte disponibili
    
    C:\zpaqBackupTest>zpaqBackup.exe archive=r:\prova\copia_zarc.zpaq -list -all >listall.txt
    .zpaq does not exist
    sorry, forgot to put a dash in front of the parameter

    ​zpaqBackup.exe -archive=r:\prova\copia_zarc.zpaq -list -all >listall.txt
    zpaqBackup.exe -archive=r:\prova\copia_zarc.rd.zpaq -list -all >listall.rd.txt

  27. #2635
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts
    First test

    Code:
    R:\prova>c:\zpaqfranz\660\zpaqlist.exe l copia_zarc.zpaq -all -out 2.txt
    11/12/2020 19:37:37 zpaqlist franz23
    11/12/2020 19:37:37 Multipart scan 0.000000 s chunks 1
    11/12/2020 19:37:37 :all
    11/12/2020 19:37:46 OUTPUT...V 1077, F 65634, 5.845.336.904 bytes 554 blocks Otime 0.000 s
    11/12/2020 19:37:47 W 010% 00006493/00064921
    11/12/2020 19:37:47 W 020% 00012986/00064921
    11/12/2020 19:37:47 W 030% 00019479/00064921
    11/12/2020 19:37:48 W 040% 00025972/00064921
    11/12/2020 19:37:48 W 050% 00032465/00064921
    11/12/2020 19:37:49 W 060% 00038958/00064921
    11/12/2020 19:37:49 W 070% 00045451/00064921
    11/12/2020 19:37:49 W 080% 00051944/00064921
    11/12/2020 19:37:50 W 090% 00058437/00064921
    11/12/2020 19:37:50 Output 182.744.963 bytes / 27.525.696 lines in 4.250 s
    11/12/2020 19:37:51 T=14.391s (all OK)

    Code:
    R:\prova>c:zpaqBackup.exe -archive=copia_zarc.rd.zpaq -list -all >1.txt
    
    Load archive time: 00:05.015
    Total time: 00.00:15.912

    Code:
    R:\prova>c:\zpaqfranz\660\zpaqlist.exe l copia_zarc.rd.zpaq
    11/12/2020 19:38:33 zpaqlist franz23
    11/12/2020 19:38:33 Multipart scan 0.000000 s chunks 1
    11/12/2020 19:38:33 :default
    Bad JIDAC header size: 3997

  28. #2636
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    42
    Thanks
    4
    Thanked 7 Times in 6 Posts
    Quote Originally Posted by fcorbelli View Post
    First test
    Code:
    R:\prova>c:\zpaqfranz\660\zpaqlist.exe l copia_zarc.rd.zpaq
    11/12/2020 19:38:33 zpaqlist franz23
    11/12/2020 19:38:33 Multipart scan 0.000000 s chunks 1
    11/12/2020 19:38:33 :default
    Bad JIDAC header size: 3997
    in .rd c/block usize=3997 bytes (4096 with header + sha1), in zpaq usize=8 bytes.. in source code zpaq715 dump error if usize < 8 only

    fix for zpaqlist.cpp

    if (os.size() < 8) {
    printf( "Bad JIDAC header size: %d\n", list_size(os));
    isbreak=true;
    if (*errors) ++*errors;
    }




    also in .rd there is a new type of block z/block, it is desirable to skip such blocks so that there are no messages about unknown blocks

  29. #2637
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    42
    Thanks
    4
    Thanked 7 Times in 6 Posts
    my test with archive 508 files

    Click image for larger version. 

Name:	tz1M1N8tTdOGllARMKl5iQ.png 
Views:	38 
Size:	233.1 KB 
ID:	8170

  30. #2638
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts
    OK, this is my (almost) complete zpaqfranz
    (not checked compiling on *nix, only Windows)

    Need some rework on VSS (stripped down C++ streams, too big after linking).
    http://www.francocorbelli.it/zpaqfranz.exe
    http://www.francocorbelli.it/zpaqfranz32.exe

    Not tested at all, but with (almost) everything I need

    zpaqfranz v7.15-franz45 journaling archiver, compiled Dec 13 2020

    Differences from ZPAQ 7.15
    Changed behaviors:
    0) Output is somewhat different (-pakka for alternative)
    1) During add() zpaqfranz stores by default the CRC-32 of the files
    This can disabled by -crc32 switch
    2) Add() using -checksum will store SHA1 hash for every file,
    doing a CRC-32 check too
    3) By default every .XLS file is forcibily added (check of datetime
    is not reliable for ancient XLS to detect changes). Disabled by -xls
    New functions:
    4) Using -comment something is possible to add ASCII text to the versions
    in add() and list(). WARNING: NO duplicates check is done
    5) In list() using -find pippo filter like |grep -i find
    In list() -comment / -comment pippo / -comment -all
    6) New command t (test) for archive test. -force for filesystem post-check
    -verbose
    7) New command p (paranoid test). Need LOTS of RAM and painfully slow. Check
    almost everything. -force -verbose
    8) New command s (size). Get the cumulative size of one or more directory.
    Useful on *nix, where it is not very quick to get this infos. Skip .zfs
    On *nix return (a kind of) free disk space on filesystem(s)
    9) New command sha1 (hashing). Calculate hash of something (default SHA1)
    -sha256, -crc32, -crc32c (HW SSE if possible), xxhash
    10) New command dir. Something similar to dir command (for *nix)
    Switches /s /a /os /od. Show cumulative size in the last line!
    11) New command help. -he show some examples, -diff differences from 7.15
    12) -noeta. Do not show ETA (for batch file)
    13) -pakka. Alternative output (for ZPAQ's GUI PAKKA)
    14) -verbose. Show more info on files
    15) -zfs. Skip ZFS's snapshots
    16) -noqnap. Skip special directories
    17) -nowindows. Turn off metafiles and system volume information
    18) -always files. Always add some file
    19) -nosort. Do not sort before adding files
    20) -nopath. Extract in local path
    21) -vss. On Windows (if running with admin power) take a snapshot (VSS) of C:
    drive before add. Useful for backup entire user profiles
    22) -find something. Find text in full filename (ex list)
    23) -replace something. Replace a -find text (manipulate output)



    Essentially, the CRC-32 of each fragment is calculated (not cleverly, actually), and then recombined to form the CRC-32 of the whole files. They are then stored in my extension (backwards compatible, I hope) of the attributes.
    There is no noticeable slowdown (lots of recalc running, but... it's a feature, not a bug!)

    Using -checksum it is possible to force storage of SHA1 codes and to recalculate CRC-32s from files to disk. In this way, in practice, a check is obtained that the "whole" CRC-32 is consistent.
    ZPAQ in its test does not (tests every single fragment with SHA1).

    The -comment option adds (very crude) ASCII text to versions, and they can be used for the extraction phase, via a fake file I hope compatible with 7.15.

    The dir command is a LONG waited dir clone (I use almost always Unix servers).

    Any comments and tests are welcome.

    Once the functions are "freezed", I will move on debugging and hardening.
    Attached Files Attached Files

  31. Thanks:

    Mike (14th December 2020)

  32. #2639
    Member
    Join Date
    Apr 2019
    Location
    Hungary
    Posts
    4
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Might be stupid question but is there a way to turn off to not store the full path? Cause if i extract something it literally creates the folders for the path. Like: Creates a C folder inside that a Users floder and etc etc...

  33. #2640
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    517
    Thanks
    25
    Thanked 45 Times in 37 Posts
    Quote Originally Posted by danieldeak70 View Post
    Might be stupid question but is there a way to turn off to not store the full path? Cause if i extract something it literally creates the folders for the path. Like: Creates a C folder inside that a Users floder and etc etc...
    -nopath

Page 88 of 89 FirstFirst ... 387886878889 LastLast

Similar Threads

  1. ZPAQ self extracting archives
    By Matt Mahoney in forum Data Compression
    Replies: 31
    Last Post: 17th April 2014, 04:39
  2. ZPAQ 1.05 preview
    By Matt Mahoney in forum Data Compression
    Replies: 11
    Last Post: 30th September 2009, 05:26
  3. zpaq 1.02 update
    By Matt Mahoney in forum Data Compression
    Replies: 11
    Last Post: 10th July 2009, 01:55
  4. Metacompressor.com benchmark updates
    By Sportman in forum Data Compression
    Replies: 79
    Last Post: 22nd April 2009, 04:24
  5. ZPAQ pre-release
    By Matt Mahoney in forum Data Compression
    Replies: 54
    Last Post: 23rd March 2009, 03:17

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •