Page 86 of 86 FirstFirst ... 3676848586
Results 2,551 to 2,569 of 2569

Thread: zpaq updates

  1. #2551
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    367
    Thanks
    12
    Thanked 40 Times in 32 Posts
    After a bit of digging the answer for this piece of code is...

    string append_path(string a, string b) {
    int na=a.size();
    int nb=b.size();
    #ifndef unix
    if (nb>1 && b[1]==':') { // remove : from drive letter
    if (nb>2 && b[2]!='/') b[1]='/';
    else b=b[0]+b.substr(2), --nb;
    }
    #endif
    if (nb>0 && b[0]=='/') b=b.substr(1);
    if (na>0 && a[na-1]=='/') a=a.substr(0, na-1);
    return a+"/"+b;
    }

    ...the extraction of all versions, with x -all, to create different paths numbered progressively.
    I think I will correct my "franz28" version of zpaq in the future to support this functionality.

    I add an updated version of PAKKA which I am testing with rather large archives (5M + files, 700GB + size),
    command to extract from version x to y, check instead extract, test everything.

    Improved UTF8 filename list, with double click (in the log tab) to directly search (~ 4/8 seconds for ~2M nodes tree)
    Last edited by fcorbelli; 25th September 2020 at 21:13.

  2. #2552
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,982
    Thanks
    298
    Thanked 1,309 Times in 745 Posts
    Google complained, had to remove attachment from this post: https://www.virustotal.com/gui/file/...69212c/details

    Also it does seem to have some inconvenient things, like accessing "https://www.francocorbelli.com:443/konta.php?licenza=PAKKA&versione=PAKKA.DEMO".

  3. #2553
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    367
    Thanks
    12
    Thanked 40 Times in 32 Posts
    Quote Originally Posted by Shelwien View Post
    Google complained, had to remove attachment from this post: https://www.virustotal.com/gui/file/...69212c/details

    Also it does seem to have some inconvenient things, like accessing "https://www.francocorbelli.com:443/konta.php?licenza=PAKKA&versione=PAKKA.DEMO".
    1) this is not a virus (but a MPRESS packed file with some .EXE as resources, as usual for Delphi code)
    starting from 2013 (www.francocorbelli.it/pakka)
    you can see here
    http://www.francocorbelli.it/nuovosito/vari.html

    In other words, it is a monolithic EXE that extracts executable programs from its resources (in the% temp% \ pkz folder) so as not to depend on anything and to be portable.
    However, since directly linked programs (zpaq custom executables) may be obsolete, it has a mechanism that offers to download them directly (windows update style),
    useful for debug
    Click image for larger version. 

Name:	update.jpg 
Views:	24 
Size:	19.8 KB 
ID:	7921


    zpaq RCDATA zpaqfranz.exe
    zpaq32 RCDATA zpaqfranz32.exe
    testa RCDATA testa.exe
    sortone RCDATA sortone.exe
    sortone32 RCDATA sortone32.exe
    codepage RCDATA codepage.exe

    Those are
    - my zpaq.cpp patched 64 bit (source already posted, here it is
    http://www.francocorbelli.it/pakka/zpaqfranz/)
    - my zpaq.cpp patched 32 bit (source already posted)
    - my software like "head" (to refresh filesize when zpaq is running)
    - sortone my delphi 64-bit special sorter (for zpaq output)
    - sortone32 same for 32 bit
    - codepage my C software to set codepage UTF-8 into Windows's shell (to restore UTF8-file)

    Code:
    program testa;
    
    {$APPTYPE CONSOLE}
    
    uses
      SysUtils,classes;
    
    var
      filename:string;
      Stream: TFileStream;
      Value: char;
    function prendiDimensioneFile(i_nomefile:string):int64;
    var
       F:file of byte;
      SearchRec : TSearchRec;
    begin
       Result:=0;
      if FindFirst(i_nomefile, faAnyFile, SearchRec ) = 0 then                  // if found
       Result := Int64(SearchRec.FindData.nFileSizeHigh) shl Int64(32) +    // calculate the size
                   Int64(SearchREc.FindData.nFileSizeLow);
      sysutils.FindClose(SearchRec);
    end;
    
    begin
      { TODO -oUser -cConsole Main : Insert code here }
       if ParamCount<>1 then
       begin
           Writeln('Testa 1.0 - by Franco Corbelli');
           Writeln('Need just 1 parameter (filename)');
           Exit;
       end;
       filename:=ParamStr(1);
       if not FileExists(filename) then
       begin
           Writeln('File name does not exists '+filename);
           Exit;
       end;
       Stream := TFileStream.Create(FileName, fmOpenRead or fmShareDenyNone);
       try
           Stream.ReadBuffer(Value, SizeOf(Value));//read a 4 byte integer
    ///       writeln('#1 '+inttostr(Integer(value)));
       except
       ///    writeln('Except');
       end;
       Stream.Free;
       Writeln(prendidimensionefile(filename));
    end.
    Code:
    program sortone;
    
    {$APPTYPE CONSOLE}
    
    {$R *.res}
     {$I defines.inc}
    
    
    uses
    system.classes,system.sysutils;
    
    
    var
      gf_start:integer;
      gf_version:integer;
    
    
    function miacompara(List: TStringList; Index1, Index2: Integer): Integer;
    var
      i:integer;
      s1,s2:string;
    begin
      s1:=list[index1].Substring(gf_start);//+list[index1].Substring(42,4);
      s2:=list[index2].Substring(gf_start);//+list[index1].Substring(42,4);
    
      if s1=s2 then
      begin
        /// stessa porzione, sortiamo per parte iniziale
        s1:=list[index1].Substring(gf_version);
        s2:=list[index2].Substring(gf_version);
    
        if s1=s2 then
        begin
          result:=0;
        end
        else
        if s1<s2 then
          result:=-1
        else
        result:=1;
      end
    
      else
      if s1<s2 then
        result:=-1
      else
        result:=1;
    
    end;
    
    var
      sl:tstringlist;
      inizio:tdatetime;
      i:integer;
      filename:string;
      outputfile:string;
      totale:tdatetime;
    begin
      try
        { TODO -oUser -cConsole Main : Insert code here }
      except
        on E: Exception do
          Writeln(E.ClassName, ': ', E.Message);
      end;
    
      gf_start:=0;
      if paramcount<>4 then
      begin
        writeln('Sortone V1.1 - 64 bit');
        writeln('4 parameters filename version path outputfile');
        writeln('Example z:\1.txt 42 47 z:\2.txt');
        exit;
      end;
    
      filename:=paramstr(1);
      if not fileexists(filename) then
      begin
          writeln('File not found '+filename);
          exit;
      end;
    
      try
        gf_version:=strtointdef(paramstr(2),0);
      finally
      end;
    
      if gf_version=0 then
      begin
        writeln('Strange version start');
        exit;
      end;
    
    
      try
        gf_start:=strtointdef(paramstr(3),0);
      finally
      end;
    
      if gf_start=0 then
      begin
        writeln('Strange column start');
        exit;
      end;
    
      outputfile:=paramstr(4);
      if fileexists(outputfile) then
        deletefile(outputfile);
      if fileexists(outputfile)  then
      begin
        writeln('We have a immortal '+outputfile);
        exit;
      end;
    
      sl:=tstringlist.create;
    
      totale:=now;
      inizio:=now;
      writeln(timetostr(now)+' load/column  '+inttostr(gf_start));
      sl.loadfromfile(filename);
      writeln(timetostr(now)+' end load in  '+floattostr((now-inizio)*100000));
    
      inizio:=now;
      writeln(timetostr(now)+' purge');
    
      for i:=sl.Count-1 downto 0 do
        begin
        if sl[i]='' then
        begin
          sl.Delete(i);
        end
        else
        begin
          if sl[i][1]<>'-' then
            sl.Delete(i);
        end;
        end;
    
      writeln(timetostr(now)+' end purge in '+floattostr((now-inizio)*100000));
      writeln(timetostr(now)+' lines/sort   '+inttostr(sl.Count-1));
    
      inizio:=now;
      sl.CustomSort(miacompara);
      writeln(timetostr(now)+' end sort in  '+floattostr((now-inizio)*100000));
      inizio:=now;
      sl.SaveToFile(outputfile);
      writeln(timetostr(now)+' end save in  '+floattostr((now-inizio)*100000));
      writeln(timetostr(now)+' total time   '+floattostr((now-totale)*100000));
    
    end.
    Code:
    /*
    gcc -O3 codepage.c -o codepage.exe
    */
    #include <stdio.h>
    #include <windows.h>
    #define str_to_int(str) strtoul(str, (TCHAR **) NULL, 10)
                               
    int main(int argc, char * argv[])
    {
        UINT in_cp;
        UINT out_cp;
        in_cp=65001;
        out_cp=65001;
        SetConsoleCP(in_cp);
        SetConsoleOutputCP(out_cp);
        in_cp=GetConsoleCP();
        out_cp=GetConsoleOutputCP();
        printf("CodePage in=%u out=%u\n", in_cp, out_cp);
        return 0;
    }

    2) as previously stated, it's a form (a Delphi-form made a separate EXE with $ifdef and so on) of my little ERP (with it's own commercial license).
    This one
    Click image for larger version. 

Name:	geo.jpg 
Views:	33 
Size:	196.0 KB 
ID:	7920
    http://www.francocorbelli.it/nuovosito/zarc.html

    In this case, of course, I have stripped the "real" time license (briefly if you do not pay every year, you do not get updates) for a free one that is "always good".
    If it's a problem, I can modify the code to turn it off (a lot of $ifdef required, but doable).

    As you can see on the first run you can download the updates directly from my site
    Code:
    if frmMainPakka.GetInetFile('http://www.francocorbelli.it/jita/'+extractfilename(i_nomefile), filetemp) then
           Result:=CopyFile(PChar(filetemp),PChar(i_nomefile),false);
    Obviously in this case I could log the applicant's IP, maybe from the web server.
    But who cares?

    Should I turn it off altogheter?


    3) this is the currently build with and without mpress

    http://www.francocorbelli.it/pakka/mpressed.exe
    SHA1
    d46996e1d265a94ea3f0e439d2de3328db71135a

    http://www.francocorbelli.it/pakka/not-mpressed.exe
    SHA1
    3294a876f43be64d0a9567dbf39a873bab27e850

    4) Into the delphi there is a function then, maybe, the euristic antivirus do not like very much.
    It's about "kill-every-file-within-filemask-in-a-folder"

    Code:
    procedure SterminaFileConMaschera(i_path:string;i_maschera:string);
    var    elencofile:tstringlist;
           i:integer;
           nomefile:TStringList;
    begin
       if i_path='' then exit;
       if not saggiascrivibilitacartella(i_path) then exit;
    
           elencofile:=tstringlist.create;
           nomefile:=tstringlist.create;
           enumerafile(i_path,i_maschera,elencoFile,nomefile);
           for i:=0 to elencofile.count-1 do
               if fileexists(elencofile.strings[i]) then
               begin
                   ///toglireadonly(elencofile.strings[i]);
                   deletefile(pchar(elencofile.strings[i]));
               end;
           elencofile.free;
           nomefile.free;
    end;
    And used like this
    Code:
    procedure TfrmMainPakka.stermina;
    begin
       SterminaFileConMaschera(GetTempDirectory,'*.txt');
       SterminaFileConMaschera(GetTempDirectory,'*.bat');
       SterminaFileConMaschera(GetTempDirectory,'*.bin');
    end;
    In fact, delete the temporary files extracted into
    %temp%\pkz


    5) There is a "strange" function too (from virus euristic detector),
    a "language checker"
    Code:
    function g_getLanguage:string;
    var
      wLang : LangID;
      szLang: Array [0..254] of Char;
    begin
      wLang := GetSystemDefaultLCID;
      VerLanguageName(wLang, szLang, SizeOf(szLang));
      Result:=szLang;
    end;
    function isItaliano:boolean;
    begin
       result:=pos('italia',lowercase(g_getlanguage))>0
    end;
    That, for default, turn on Italian strings if... running on Italy's Windows.
    Otherwise turn on english (not fully translated in fact, it's heavy and boring)

    6) If you run as an admin, you get another "risky" function
    Click image for larger version. 

Name:	admin.jpg 
Views:	25 
Size:	40.5 KB 
ID:	7922

    Register the ZPAQ extension to the software, so you can "double click" and open directly
    the file.
    Maybe the google anti virus do not like it very much, I do not know.



    I hope I've been exhaustive.
    Last edited by fcorbelli; 25th September 2020 at 17:05. Reason: 0JzQvtC5INGA0YPRgdGB0LrQuNC5INC90LUg0L7Rh9C10L3RjCDRhdC+0YDQvtGI0L4=

  4. Thanks:

    Shelwien (25th September 2020)

  5. #2554
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,982
    Thanks
    298
    Thanked 1,309 Times in 745 Posts
    Thanks for description, but hosting executables here is still risky - google has a lot of false positives and easily blocks sites in chrome.

  6. #2555
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    367
    Thanks
    12
    Thanked 40 Times in 32 Posts
    I apologize for this problem.
    I just removed all (2020+) uploaded executables.
    I will not post anymore compiled code

  7. #2556
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    367
    Thanks
    12
    Thanked 40 Times in 32 Posts
    The version 29 of my ZPAQ "fork" includes

    0) changed the -donotusexls to -xls (too long to remember)
    1) minor fix in forced addition of XLS files. Excludes $data
    2) show size and quantity of forcibly added XLS files (in human readable style) after the summary
    Code:
    (...)
    270.103745 + (17.547333 -> 0.000000 -> 0.000000) = 270.103745 MB
    Forced XLS has included 17.547.333 bytes in 143 files

    3) with the verbose option indicates which they are
    Code:
    ...
    ENFORCING XLS c:/zpaqfranz/standard/trattore2.xls
    ENFORCING XLS c:/zpaqfranz/standard/vittoria.xlsx
    ENFORCING XLS c:/zpaqfranz/standard/wind.xls
    ENFORCING XLS c:/zpaqfranz/standard/xerox.xls
    Adding 17.547333 MB in 143 files -method 14 -threads 12 at 2020-09-29 16:59:55.
    0 +added, 0 -removed.7.333 of 17.547.333 0/sec
    truncating archive from 270103849 to 270103745
    ...
    4) fixed, I hope definitively, the "strange" behavior of the append_path function that I have splitted into two.
    One (Matt's append_path() ) which remains unchanged, the other myappend_path() which is called by the list () function via a flag in function's call.
    Very quick, and very dirty.

    Code:
    string append_path(string a, string b) {
      int na=a.size();
      int nb=b.size();
    #ifndef unix
      if (nb>1 && b[1]==':') {  // remove : from drive letter
        if (nb>2 && b[2]!='/') b[1]='/';
        else b=b[0]+b.substr(2), --nb;
      }
    #endif
      if (nb>0 && b[0]=='/') b=b.substr(1);
      if (na>0 && a[na-1]=='/') a=a.substr(0, na-1);
      return a+"/"+b;
    }
    string myappend_path(string a, string b) {
      return a+"|"+b;
    }

    Code:
      int64_t read_archive(const char* arc, int *errors=0, int i_myappend=0);  // read arc
    Code:
    // List contents
    int Jidac::list() {
    
      // Read archive into dt, which may be "" for empty.
      int64_t csize=0;
      int errors=0;
      
      if (archive!="") csize=read_archive(archive.c_str(),&errors,1); /// AND NOW THE MAGIC ONE!
    Code:
    ...
    if (i_myappend)
    {
      fn=myappend_path(itos(ver.size()-1, all), fn);
    }
    else
    {
       fn=append_path(itos(ver.size()-1, all), fn);
     }
    ...
    No function pointer, I brutally prefer a flag.

    In this way it is possible to extract the versions massively (extract -all),
    just like 7.15, and to have an enumeration (list -all) that does not eliminate the fundamental character ":" of the Windows path for use with a program that provides the GUI (PAKKA, for example).

    I don't attach any executable code
    Attached Files Attached Files

  8. #2557
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    367
    Thanks
    12
    Thanked 40 Times in 32 Posts
    I attach the source of a small modification I made to zpaq 6.60, transforming into... zpaqlist.
    Essentially I removed various functions, except the list (), which I changed according to my needs.
    Which are to get a "good" list of files sorted by file NAME, and not by version (as ZPAQ 7.15 does).

    So with
    Code:
    zpaqlist list r:\secundo.zpaq -all >z:\a.txt
    the output will be
    Code:
    zpaqlist v6.60-franz5 file lister, compiled Oct  7 2020
    r:/secundo.zpaq: 23 versions, 1831 files, 9444 fragments, 270.103745 MB
    |      1 2020-09-29 15:50:01
    |      2 2020-09-29 15:53:12
    |      3 2020-09-29 15:55:10
    |      4 2020-09-29 15:56:31
    |      5 2020-09-29 15:57:38
    |      6 2020-09-29 15:58:20
    |      7 2020-09-29 15:58:42
    (...)
    - 000001 2020-09-29 15:32:07              0 c:/nz/
    (...)
    - 000001 2020-09-29 15:47:40         142539 c:/zpaqfranz/zpaq.cpp
    - 000002 2020-09-29 15:52:40         142543 c:/zpaqfranz/zpaq.cpp
    - 000003 2020-09-29 15:54:35         142599 c:/zpaqfranz/zpaq.cpp
    - 000004 2020-09-29 15:55:57         142640 c:/zpaqfranz/zpaq.cpp
    - 000006 2020-09-29 15:58:09         142636 c:/zpaqfranz/zpaq.cpp
    - 000008 2020-09-29 16:00:31         142636 c:/zpaqfranz/zpaq.cpp
    - 000009 2020-09-29 16:03:03         142758 c:/zpaqfranz/zpaq.cpp
    - 000010 2020-09-29 16:06:16         142671 c:/zpaqfranz/zpaq.cpp
    - 000011 2020-09-29 16:09:13         142886 c:/zpaqfranz/zpaq.cpp
    - 000012 2020-09-29 16:10:13         142886 c:/zpaqfranz/zpaq.cpp
    - 000016 2020-09-29 16:19:12         143009 c:/zpaqfranz/zpaq.cpp
    - 000017 2020-09-29 16:20:34         143012 c:/zpaqfranz/zpaq.cpp
    - 000019 2020-09-29 16:27:55         143053 c:/zpaqfranz/zpaq.cpp
    - 000021 2020-09-29 16:31:54         143043 c:/zpaqfranz/zpaq.cpp
    - 000023 2020-09-29 16:48:10         143115 c:/zpaqfranz/zpaq.cpp

    I have tried to "transplant" the portion of code that interests me (my new mylist () function) from 6.60 to 7.15, but I'm having trouble.

    Is there anyone who knows data structures (and may be C++) better than me who can help ?

    Otherwise I will have to study a lot of the original code which is not exactly written in my favorite style (aka: a mess).

    Thank you

    This is mylist() [a patchwork with original code, not really elegant...]

    Code:
    int Jidac::mylist() 
    {
      int64_t csize=0;
      if (archive!="") 
      {
        csize=read_archive();
        if (csize==0) exit(1);
      }
    
    
    // Mark files in dt with written=0 selected by files, onlyfiles, and notfiles 
       for (DTMap::iterator p=dt.begin(); p!=dt.end(); ++p) 
      {
        if (p->second.dtv.size()<1) 
        {
          fprintf(stderr, "Invalid index entry: %s\n", p->first.c_str());
          error("corrupted index");
        }
        if (isselected(p->first.c_str())
            && p->second.dtv.size() 
            && (all || p->second.dtv.back().date))
          p->second.written=0;
      }
      
    
    /// versions
      for (int i=since; i<size(ver); ++i) 
      {
        if (i==0 && ver[i].updates==0
            && ver[i].deletes==0 && ver[i].date==0 && ver[i].usize==0)
          continue;
        fprintf(con, "| %6d %s\n", i,dateToString(ver[i].date).c_str());
      }
    
      
      
      vector<DTMap::const_iterator> filelist;
      for (DTMap::const_iterator p=dt.begin(); p!=dt.end(); ++p)
        if (p->second.written==0)
          filelist.push_back(p);
    
      for (unsigned fi=0; fi<filelist.size(); ++fi) 
      {    
        DTMap::const_iterator p=filelist[fi];
        for (unsigned i=0; i<p->second.dtv.size(); ++i) 
        {
            if (p->second.dtv[i].version>=1 //since
              && (all || (i+1==p->second.dtv.size() && p->second.dtv[i].date))) 
            {
                if (!strstr(p->first.c_str(), ":$DATA")) /// fuck off Windows garbage
                {
                    fprintf(con, "- %06d ", p->second.dtv[i].version);
                    if (p->second.dtv[i].date) 
                    {
                        fprintf(con, "%s %14.0f ",
                            dateToString(p->second.dtv[i].date).c_str(),
                            double(p->second.dtv[i].size)
                            );
                    }
                    else 
                    {
                        fprintf(con, "%-35s", "Deleted");
                    }
                    printUTF8(p->first.c_str(), con);
                    fprintf(con, "\n");
                }
            }
        }
      }
      return 0;
    }
    PS no EXE code, and not .h header (original ZPAQ 6.60, not allowed here)
    Attached Files Attached Files

  9. #2558
    Member
    Join Date
    Oct 2020
    Location
    London
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I am backing up a large file every day (23GB)and only want to send the changes across an occasionally connected remote store.
    I thought zpaq might be the right tool to do this, but each numbered part of the archive made each day only grows larger than the previous day, even if only a single byte is changed in the file. So the parts are always appended? Is there a way of only getting a file with the changes? "Add" just doesn't seem to offer that possibility?
    EG
    zpaq add backup* bigfile
    backup1.zpaq 8GB
    Make small change at end of bigfile
    zpaq add backup* bigfile
    backup1.zpaq 8GB
    backup2.zpaq 8.1GB

    Am I missing something on how to get just a 0.1GB file?
    Thanks

  10. #2559
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    367
    Thanks
    12
    Thanked 40 Times in 32 Posts
    Quote Originally Posted by Sheddy View Post
    I am backing up a large file every day (23GB)and only want to send the changes across an occasionally connected remote store.
    I thought zpaq might be the right tool to do this, but each numbered part of the archive made each day only grows larger than the previous day, even if only a single byte is changed in the file. So the parts are always appended? Is there a way of only getting a file with the changes? "Add" just doesn't seem to offer that possibility?
    EG
    zpaq add backup* bigfile
    backup1.zpaq 8GB
    Make small change at end of bigfile
    zpaq add backup* bigfile
    backup1.zpaq 8GB
    backup2.zpaq 8.1GB

    Am I missing something on how to get just a 0.1GB file?
    Thanks
    zpaq a backup_???? bigfile -summary 1

  11. #2560
    Member
    Join Date
    Oct 2020
    Location
    London
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by fcorbelli View Post
    zpaq a backup_???? bigfile -summary 1
    Thanks for replying.
    zpaq is noticing changes:
    Copy /b bigfile+smallchange bigfile ...
    is a rookie windows mistake in my testing. bigfile becomes garbage doing this rather than appending.
    What was happening with the real data might have been down to fragment size change, but I can't reproduce it now.
    Sorry to have wasted your time

    Thanks

  12. #2561
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    367
    Thanks
    12
    Thanked 40 Times in 32 Posts
    Quote Originally Posted by Sheddy View Post
    Thanks for replying.
    zpaq is noticing changes:
    Copy /b bigfile+smallchange bigfile ...
    is a rookie windows mistake in my testing. bigfile becomes garbage doing this rather than appending.
    What was happening with the real data might have been down to fragment size change, but I can't reproduce it now.
    Sorry to have wasted your time

    Thanks
    In Windows you can do your little "cloud backup" with something like that
    Code:
    zpaqfranz a r:\path-to-be-copied\mybackup_????.zpaq c:\thebigfile -summary 1
    
    rsync -e "c:\cloud\ssh -i /cygdrive/c/cloud/myrsakey" -I --append --partial -r --progress -vv --chmod=a=rwx,Da+x --delete /cygdrive/r/path-to-be-copied myremoteuser@mybackupsite.francocorbelli.it:/myremotepath
    On *nix
    Code:
    /usr/local/bin/rsync  -I --append --omit-dir-times --no-owner --no-perms --partial --progress -e "/usr/bin/ssh -p $PORT -i $KEY "  -rlt  --delete "/zroot/folder-to-be-copied" "$USER@$SERVER:/home/somewhereremote"

    You don't need to recreate the single ZPAQ file, you can keep the various files separate.
    ZPAQ is able to understand these cases and restore directly

    On Windows you can even try my little software
    http://www.francocorbelli.it/pakka.exe
    (currently 3/4 english, 1/4 italian, I'm working on the translation)
    that understand multipart ZPAQ file
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	multi.jpg 
Views:	10 
Size:	98.7 KB 
ID:	7973   Click image for larger version. 

Name:	multi2.jpg 
Views:	10 
Size:	20.0 KB 
ID:	7974   Click image for larger version. 

Name:	multi3.jpg 
Views:	11 
Size:	169.2 KB 
ID:	7975  

  13. #2562
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    367
    Thanks
    12
    Thanked 40 Times in 32 Posts
    Quote Originally Posted by Sheddy View Post
    Thanks for replying.
    zpaq is noticing changes:
    Copy /b bigfile+smallchange bigfile ...
    is a rookie windows mistake in my testing. bigfile becomes garbage doing this rather than appending.
    What was happening with the real data might have been down to fragment size change, but I can't reproduce it now.
    Sorry to have wasted your time

    Thanks
    In Windows you can do your little "cloud backup" with something like that
    Code:
    zpaqfranz a r:\path-to-be-copied\mybackup_????.zpaq c:\thebigfile -summary 1
    
    rsync -e "c:\cloud\ssh -i /cygdrive/c/cloud/myrsakey" -I --append --partial -r --progress -vv --chmod=a=rwx,Da+x --delete /cygdrive/r/path-to-be-copied myremoteuser@mybackupsite.francocorbelli.it:/myremotepath
    On *nix
    Code:
    /usr/local/bin/rsync  -I --append --omit-dir-times --no-owner --no-perms --partial --progress -e "/usr/bin/ssh -p $PORT -i $KEY "  -rlt  --delete "/zroot/folder-to-be-copied" "$USER@$SERVER:/home/somewhereremote"

    If you have a remote server (Linux or Unix) I suggest to verify the BIGFILE with something like that

    On Windows
    Code:
    c:\cloud\sha1deep -e c:\parte\BIGFILE >>r:\test\cloud\sha1.txt
    
    (...do the ZPAQ...)
    
    SETLOCAL
    SET CWRSYNCHOME=c:\cloud
    
    REM the key is here (arrrgghhhhh!!!!)
    SET PATH=%CWRSYNCHOME%\BIN;%PATH%
    c:\cloud\ssh -p 22 -i c:\cloud\myremotekey myuser@myremoteserver /tank/estratto/checkfranco.sh

    On the remote (in this case Unix /tank/estratto/checkfranco.sh script)

    Code:
    rm -r /tank/estratto/c:
    /tank/estratto/zpaq x "/tank/prove/cloud/provona_???????.zpaq" -to /tank/estratto -summary 1 -force
    sha1deep /tank/estratto/c:/parte/BIGFILE >/tank/estratto/remoto.txt
    tail -1 /tank/prove/cloud/sha1.txt >>/tank/estratto/remoto.txt
    /usr/local/bin/smtp-cli -verbose -server=mail.something.com -4 --port 587 -user=mylogg@something -pass=mygoodpassword -from=mylogg@something.com -to logg@something.com -subject  "BIGFILE on BSD" -body-plain=/tank/estratto/remoto.txt


    In other words

    (Windows)
    - calculate the SHA1 or MD5 hash of the large file
    - write the hash to a file in the same folder
    - do the ZPAQ
    - synchronize with rsync on a remote server, typically with --append
    - run a script on the remote server that extracts the large file from the ZPAQ file, recalculates the SHA1, append to previous "windows" file, and sends it to you by email.

    Getting something like that
    d196ec95accbeed7377d3f141eecaa5e66a1af35 /tank/estratto/c:/parte/BIGFILE
    d196ec95accbeed7377d3f141eecaa5e66a1af35 c:\parte\BIGFILE

    Now you are 100% sure that "BIGFILE" in c:\parte (Windows) folder == to the BIGFILE
    extracted from ZPAQ sequence, and your backup is completely reliable.
    If not, something went wrong.

    To rsync and run scripts on the remote server, you will normally use ssh tunnels with key authentication

    When using single ZPAQ it's necessary to use --append, with much smaller ???? it depends.

    This is a ~1TB example with big files

    Code:
    -rw-r--r--  1 theuser  theuser   288G Oct 10 19:30 fserver_condivisioni.zpaq
    -rw-r--r--  1 theuser  theuser   384G Oct 10 21:01 macchinavirtuale.zpaq
    -rw-r--r--  1 theuser  theuser   771B Oct  9 06:18 md5.txt
    -rw-r--r--  1 theuser  theuser   108G Oct  9 09:11 nonlosappiamo.zpaq
    -rw-r--r--  1 theuser  theuser   1.6M Oct  8 19:31 v-server_appoggio.zpaq
    -rw-r--r--  1 theuser  theuser    22G Oct 10 19:31 v-server_bus.zpaq
    -rw-r--r--  1 theuser  theuser    96M Oct  8 19:31 v-server_condivisioni.zpaq
    -rw-r--r--  1 theuser  theuser    50G Oct 10 19:41 v-server_dump.zpaq
    -rw-r--r--  1 theuser  theuser   7.3G Oct 10 19:31 v-server_merak.zpaq
    And this one (a single 30GB file) with "???"
    Code:
    -rwxrwxrwx  1 theuser  theuser  31155130921 May 15 19:12 provona_0000001.zpaq
    -rwxr-xr-x  1 theuser  theuser     10885693 May 16 19:48 provona_0000002.zpaq
    -rwxr-xr-x  1 theuser  theuser     66240169 May 16 19:48 provona_0000003.zpaq
    -rwxr-xr-x  1 theuser  theuser      4112652 May 16 19:48 provona_0000004.zpaq
    -rwxr-xr-x  1 theuser  theuser      1790026 May 16 19:48 provona_0000005.zpaq
    -rwxr-xr-x  1 theuser  theuser      1790026 May 16 19:48 provona_0000006.zpaq
    -rwxr-xr-x  1 theuser  theuser   1752477789 May 16 20:19 provona_0000007.zpaq
    -rwxr-xr-x  1 theuser  theuser      8564799 May 17 17:20 provona_0000008.zpaq
    -rwxr-xr-x  1 theuser  theuser    217784380 May 27 14:32 provona_0000009.zpaq
    -rwxr-xr-x  1 theuser  theuser     56561809 May 29 16:54 provona_0000010.zpaq
    -rwxr-xr-x  1 theuser  theuser     25770083 May 31 12:47 provona_0000011.zpaq
    -rwxr-xr-x  1 theuser  theuser      6263890 May 31 15:41 provona_0000012.zpaq
    -rwxr-xr-x  1 theuser  theuser      4812340 May 31 17:37 provona_0000013.zpaq
    -rwxr-xr-x  1 theuser  theuser    379547906 Jun  3 19:40 provona_0000014.zpaq
    -rwxr-xr-x  1 theuser  theuser   2131661943 Jun  8 11:15 provona_0000015.zpaq
    -rwxr-xr-x  1 theuser  theuser     17609985 Jun 13 13:14 provona_0000016.zpaq
    -rwxr-xr-x  1 theuser  theuser    204799077 Jun 16 18:36 provona_0000017.zpaq
    -rwxr-xr-x  1 theuser  theuser     15621334 Jun 21 14:45 provona_0000018.zpaq
    -rwxr-xr-x  1 theuser  theuser     12493675 Jun 23 18:08 provona_0000019.zpaq
    -rwxr-xr-x  1 theuser  theuser      3627665 Jun 24 12:08 provona_0000020.zpaq
    -rwxr-xr-x  1 theuser  theuser     47595439 Jun 29 15:04 provona_0000021.zpaq
    -rwxr-xr-x  1 theuser  theuser     91068244 Jul  3 12:19 provona_0000022.zpaq
    -rwxr-xr-x  1 theuser  theuser      9297953 Jul  5 12:07 provona_0000023.zpaq
    -rwxr-xr-x  1 theuser  theuser          795 Jul  8 14:52 provona_0000024.zpaq
    -rwxr-xr-x  1 theuser  theuser          561 Jul  8 14:52 provona_0000025.zpaq
    -rwxr-xr-x  1 theuser  theuser          561 Jul  8 14:58 provona_0000026.zpaq
    -rwxr-xr-x  1 theuser  theuser      7127483 Jul  8 14:58 provona_0000027.zpaq
    -rwxr-xr-x  1 theuser  theuser     12877999 Jul 16 14:33 provona_0000028.zpaq
    -rwxr-xr-x  1 theuser  theuser     23301206 Jul 18 14:03 provona_0000029.zpaq
    -rwxr-xr-x  1 theuser  theuser     12154738 Jul 22 14:25 provona_0000030.zpaq
    -rwxr-xr-x  1 theuser  theuser   1819992129 Jul 25 19:00 provona_0000031.zpaq
    -rwxr-xr-x  1 theuser  theuser    605108463 Jul 26 17:00 provona_0000032.zpaq
    -rwxr-xr-x  1 theuser  theuser      1790724 Jul 31 13:35 provona_0000033.zpaq
    -rwxr-xr-x  1 theuser  theuser   1352347822 Aug 12 10:56 provona_0000034.zpaq
    -rwxr-xr-x  1 theuser  theuser   1402538403 Aug 14 16:22 provona_0000035.zpaq
    -rwxr-xr-x  1 theuser  theuser    399385264 Aug 21 17:18 provona_0000036.zpaq
    -rwxr-xr-x  1 theuser  theuser     68718448 Aug 23 10:43 provona_0000037.zpaq
    -rwxr-xr-x  1 theuser  theuser   1433663439 Aug 26 19:06 provona_0000038.zpaq
    -rwxr-xr-x  1 theuser  theuser   1015318964 Sep  4 14:33 provona_0000039.zpaq
    -rwxr-xr-x  1 theuser  theuser   1679259144 Sep  7 17:13 provona_0000040.zpaq
    -rwxr-xr-x  1 theuser  theuser    184524795 Sep  8 19:16 provona_0000041.zpaq
    -rwxr-xr-x  1 theuser  theuser    343135110 Sep 13 18:50 provona_0000042.zpaq
    -rwxr-xr-x  1 theuser  theuser    563944500 Sep 15 19:27 provona_0000043.zpaq
    -rwxr-xr-x  1 theuser  theuser    618129849 Sep 21 18:07 provona_0000044.zpaq
    -rwxr-xr-x  1 theuser  theuser    183112102 Sep 24 18:10 provona_0000045.zpaq
    -rwxr-xr-x  1 theuser  theuser    781818676 Sep 26 19:29 provona_0000046.zpaq
    -rwxr-xr-x  1 theuser  theuser   1179932667 Sep 30 14:37 provona_0000047.zpaq
    -rwxr-xr-x  1 theuser  theuser    767243388 Oct  7 13:19 provona_0000048.zpaq
    -rwxr-xr-x  1 theuser  theuser     93983621 Oct  7 13:19 provona_0000049.zpaq
    -rwxr-xr-x  1 theuser  theuser   1197697832 Oct 11 18:24 provona_0000050.zpaq

  14. #2563
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    367
    Thanks
    12
    Thanked 40 Times in 32 Posts
    And this is the zpaqlist 6.60-franz9
    The output indicates the date in Italian format (dd/mm/yyyy),
    the file size is divided by thousands (with dots) and is sorted by file name (including the : if any).

    Essentially a pre-parsed format for PAKKA (or any such program) that gets about 3x faster.
    Compile with gcc 7.3.0 on both 32 and 64 bit Windows
    Versions at the beginning to simplify recognition (iniziano con |).
    No bloat lines, writes key error into output stream (not error) to catch the
    redirection >

    Code:
    zpaqlist v6.60-franz9 file lister, compiled Oct  8 2020
    r:/1.zpaq: 319 versions, 957 files, 4081 fragments, 119.742900 MB
    |      1 2019-02-25 11:34:18
    |      2 2019-02-25 15:55:00
    |      3 2019-02-25 16:28:21
    (...)
    |    318 2020-02-17 00:00:54
    |    319 2020-02-18 00:00:55
    (...)
    - 000001 05/09/2015 13:48:20 0                        z:/myzarc/1
    - 000001 07/09/2015 14:38:10 367                      z:/myzarc/1.bat
    - 000001 06/11/2018 17:49:56 341.409                  z:/myzarc/1.eml
    - 000001 17/05/2016 14:55:48 27.648                   z:/myzarc/1.xls
    - 000001 05/09/2015 13:48:20 0                        z:/myzarc/1000
    - 000001 05/09/2015 13:48:22 0                        z:/myzarc/110
    - 000001 05/09/2015 13:48:20 0                        z:/myzarc/120
    - 000001 21/10/2010 17:27:52 9.085                    z:/myzarc/17787_passing_files_from_explorer_to_delphi_app_via_dde.ZIP
    - 000001 06/11/2018 17:49:40 341.472                  z:/myzarc/2.eml
    (...)
    - 000001 23/02/2019 17:49:38 395.001                  z:/myzarc/maingeosql.pas
    - 000049 08/04/2019 12:52:32 395.150                  z:/myzarc/maingeosql.pas
    - 000059 20/04/2019 15:09:18 395.161                  z:/myzarc/maingeosql.pas
    - 000060 20/04/2019 18:23:24 403.036                  z:/myzarc/maingeosql.pas
    - 000062 21/04/2019 15:50:08 410.475                  z:/myzarc/maingeosql.pas
    - 000103 06/06/2019 14:23:56 410.585                  z:/myzarc/maingeosql.pas
    - 000125 29/06/2019 13:01:22 410.926                  z:/myzarc/maingeosql.pas
    (...)
    0.109 seconds (all OK)
    Attached Files Attached Files

  15. #2564
    Member
    Join Date
    Oct 2020
    Location
    London
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Thank you fcorbelli for the useful information. I hadn't even thought of verifying on the server side, but it is an important consideration

  16. #2565
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    367
    Thanks
    12
    Thanked 40 Times in 32 Posts
    You can always press the Thanks button under the posts

    More seriously: the problem of telematic transmission of large amounts of data is
    just my area of ​​work.
    Usually at least a double check is applied.
    Locally: the unpacking of the ZPAQ file is identical to the original data
    (and here is the patch I made to the source code for the management of the XLS files:
    the verification would fail otherwise).

    Remotely: the ZPAQ file is identical to the local one.

    In this case, for the transitive property, the saved files are correct.

    Another level is to actually unpack the remote data as well.
    This is to make sure that the compiled version of ZPAQ
    (which is different, works on Unix and not Windows, different compiler etc)
    is able to "understand" the format perfectly.
    The source is the same, but there are many #ifdefs in the source.
    Any copy that has not actually been checked out is not safe.
    In this case I suggest the use of hashdeep,
    even if it requires a lot of "sed" work to do the checks
    (it stores the entire paths).
    On machines with SSD disks, running in parallel, it has extremely
    higher performance rather than repeated sequential hash calculations.

    In case of even greater needs I just use truecrypt containers,
    which obviously contain every single bit of the filesystem
    without the possibility of error.
    Similarly in the case of vmdk virtual disks (other work environment).
    By sending a virtual disk, and compared the hash against the local,
    I don't have to check the hundreds of thousands of small files it may contains.

    The wise (storage) man does not trust, and controls.
    Then check again.
    And again.
    And one more time.
    Then restart checking.

    I forgot to specify that data is typically encrypted symmetrically,
    apart from the case of Truecrypt containers.

    Note, however, that ZPAQ has the unfortunate tendency to use large amounts of memory even when enumerating files (list).
    In some cases (very large files, say ~ 400GB compressed for many versions, say 50+) it can consume all the RAM (physical and virtual) of the PC crashing.
    Therefore it is good to make sure that you have machines with enough RAM when handling big ZPAQs.

    It's something that drove me crazy for a long time, until I carefully monitored memory consumption

    Because, as mentioned, having a ZPAQ file does not mean at all being able to resume its contents, especially if it is used in non-hobby areas

  17. Thanks (3):

    Bulat Ziganshin (19th October 2020),Shelwien (12th October 2020),Sobic (18th October 2020)

  18. #2566
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    367
    Thanks
    12
    Thanked 40 Times in 32 Posts
    This is my current zpaqlist-franz11
    Click image for larger version. 

Name:	tre.jpg 
Views:	25 
Size:	283.1 KB 
ID:	7989
    It is about 3x faster than "standard" ZPAQ 7.15 listing files (on my testbed)

    Create approximately 33% smaller output (depending on the source)
    for complex ZPAQ files (my testbed 334MB vs 498MB of 7.15)
    by not repeating the same filenames,
    "deduplicating" with"?same?"
    (almost impossible to have files with? on Windows)

    Shows the progress (extremely important for very large files accessed over network:
    ZPAQ 7.15 "sleeps" without letting you know where it is)
    http://www.francocorbelli.it/pakka/working.gif 0 Links

    There is no easy way to calculate ETA, I would have to make a smart predictor and it doesn't seem the case.

    Finally I put an optional switch -distinct to get the complete list of output files (instead of deduplicated one)

    =>

    result: last version of PAKKA is 5x faster than the first, considering all the improvements applied

    I do not think it is possible to do much better, without heavily rewriting the source of ZPAQ, maybe a certain percentage improvement can still be squeezed.
    Attached Files Attached Files

  19. #2567
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    13
    Thanks
    4
    Thanked 2 Times in 2 Posts
    Hi fcorbelli

    for zpaq list

    1 to calculate the size of files in the archive, you need to load all h blocks into memory, this is the first memory block that zpaq consumes
    - I don't know how to optimize it yet

    2 to understand that the files in the archive are duplicates, load the corresponding sets of indexes into memory and compare them by length or by element if the length of the sets is the same for the files

    - at the moment my solution is as follows, I calculate sha1 (a set of indices) for each file and then sort the list of files by key (size, hash) in descending order

  20. #2568
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    367
    Thanks
    12
    Thanked 40 Times in 32 Posts
    Quote Originally Posted by SpyFX View Post
    Hi fcorbelli

    for zpaq list...
    My current bottleneck is the output stage, the fprintf.
    With or without text buffering.
    I need to make something different, like 'sprintf' in a giant RAM buffer
    to be written in one time.
    And output running when taking data, instead of like now
    (first decode, then output).

    A very big work (I do not like very much the ZPAQ source code, I have to rewrite myself,but too time is needed)

    Even without this my zpaqlist is two time faster than anything else (on my testbed).
    Tomorrow I will post the source, and my fix for zpaq to support direct utf8 extraction without supplementary software (running on Windows).

    Now my little PAKKA (alpha stage) runs zpaqlist and zpaqfranz in a background thread, so the Delphi GUI show progression in real time, instead of running a batch file

  21. #2569
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    367
    Thanks
    12
    Thanked 40 Times in 32 Posts
    Here current versions of zpaqfranz and zpaqlist.
    In zpaq
    Code:
    SetConsoleCP(65001);
    SetConsoleOutputCP(65001);
    enable UTF-8 codepage directly on the caller shell.
    A new switch -pakka for different output

    In zpaqlist a -pakka ("verbose") and a -out filename.txt where the output goes.

    Here the 32.2 version of PAKKA (alpha stage-half italian interface)
    http://www.francocorbelli.it/pakka.exe
    It's possible to use control wheel, shift wheel and control/shift wheel to change the font
    Need more debug
    Attached Files Attached Files

  22. Thanks:

    Mike (Yesterday)

Page 86 of 86 FirstFirst ... 3676848586

Similar Threads

  1. ZPAQ self extracting archives
    By Matt Mahoney in forum Data Compression
    Replies: 31
    Last Post: 17th April 2014, 03:39
  2. ZPAQ 1.05 preview
    By Matt Mahoney in forum Data Compression
    Replies: 11
    Last Post: 30th September 2009, 04:26
  3. zpaq 1.02 update
    By Matt Mahoney in forum Data Compression
    Replies: 11
    Last Post: 10th July 2009, 00:55
  4. Metacompressor.com benchmark updates
    By Sportman in forum Data Compression
    Replies: 79
    Last Post: 22nd April 2009, 03:24
  5. ZPAQ pre-release
    By Matt Mahoney in forum Data Compression
    Replies: 54
    Last Post: 23rd March 2009, 02:17

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •