1. So, I want to create an incremental archive like so
Code:
zpaq a backup????.zpaq "files" -index backup0000.zpaq
But I have the limitation of 4GB per file (FAT32). So I split the large first archive
Code:
splits backup0001.zpaq 4000000
And then rename the chunks to backup0002, 0003 etc.

And so, the archive breaks! Can't extract anything.

I tried to re-index it, but to no avail.

How do I create an indexed incremental archive with a file size limitation?

2. Originally Posted by nanoflooder
So, I want to create an incremental archive like so
Code:
zpaq a backup????.zpaq "files" -index backup0000.zpaq
But I have the limitation of 4GB per file (FAT32). So I split the large first archive
Code:
splits backup0001.zpaq 4000000
And then rename the chunks to backup0002, 0003 etc.

And so, the archive breaks! Can't extract anything.

I tried to re-index it, but to no avail.

How do I create an indexed incremental archive with a file size limitation?

Zpaq does not have flags that allow you to limit the size of the archive

3. Originally Posted by SpyFX
Zpaq does not have flags that allow you to limit the size of the archive
There should be to be able to limit the file size

4. I have a big problem with ZPAQ 7.15 : it segfaults when compiled into linux systems that use MUSL library instead of standard libc (Alpine Linux in my case).

The problem seems related to the stack size of the library, that is limited to 80kb instead of the 8Mb (!) provided by gLibc, causing overflows during pthread-s:

https://github.com/richfelker/musl-cross-make/issues/33

Is there a workaround for the issue ?

5. Maybe compile with non-default stack size via gcc -Wl,--stack,16777216 ?
http://www.mattmahoney.net/dc/zpaq715.zip

6. Originally Posted by Shelwien
Maybe compile with non-default stack size via gcc -Wl,--stack,16777216 ?
http://www.mattmahoney.net/dc/zpaq715.zip
Unfortunately, this is not working.

Even if compiled -static, it segfaults almost immediately.

I'm a Sys-op guy, not really into compiling and coding, this is what I get running into gdb:

Starting program: /root/zpaq a test /
zpaq v7.15 journaling archiver, compiled May 3 2019
test.zpaq: 0 versions, 0 files, 0 fragments, 0.000000 MB
Updating test.zpaq at offset 0 + 0
Adding 1326.112348 MB in 42966 files -method 14 -threads 1 at 2019-05-03 12:12:34.
[New LWP 2317]
[New LWP 2318]
1.27% 0:00:27 [1..236] 16754818 -method 14,97,0

[Switching to LWP 2317]
0x00007ffff7fac5c0 in libzpaq::Predictor::Predictor(libzpaq::ZPAQL&) ()
(gdb)

(gdb) backtrace
#0 0x00007ffff7fac5c0 in libzpaq::Predictor::Predictor(libzpaq::ZPAQL&) ()
#1 0x00007ffff7fc7122 in libzpaq::compressBlock(libzpaq::StringBuffer*, libzpaq::Writer*, char const*, char const*, char const*, bool) ()
#4 0x00007ffff7fddd11 in __clone () at src/thread/x86_64/clone.s:21
Backtrace stopped: frame did not save the PC
(gdb)

(gdb) frame 3

7. Try compiling with -DNOJIT ?

8. Originally Posted by Shelwien
Try compiling with -DNOJIT ?
Still no luck on this. I think I'll give up on this Alpline Linux and try something that has glibc instead of musl.

9. Hello, I have a question about zpaq. I have a use case where different people must add directories with the same name, but differing contents to the same archive. All files that get added are necessary (we are basically deleting nothing). zpaq would be ideal for this because of the great deduplication and compression.

When different people add directories with the same name but different files, a normal extract only gives the latest set of added files because zpaq thinks the rest got deleted from the directory. That leads to my question: is there a way to use zpaq in an append-only mode (journaling which records no delete actions) which still has compression and deduplication? I know I can just use -all switch and deal with the files myself, but I was hoping for an easier way.

I also want to thank you for the great tool. I use it daily, mainly because of the deduplication. Sorry for any mistakes, I'm not a native speaker.

10. Today as an entry to this Code Golf challenge I started playing with zpaq. When I used apt-get install zpaq on an Ubuntu 18.04 virtual machine, I got zpaq version 1.10, and produced a fairly good result. Then someone pointed out that 1.10 is over ten years old, and the current version of zpaq is 7.15.

It seems that the commandline interface underwent a major change in version 2, which isn't surprising. But what is surprising:
* a few of the options from 1.10 relating to metadata do not have any equivalent win 7.15: n (don't store filenames), s (don't store SHA1 checksums), i (don't store file sizes), q (no messages)
* Even skipping the metadata problem, I'm unable to figure out what current -method options to use to reproduce the output from 1.10.

My goal in this code-golf challenge is to produce the smallest possible compressed file plus commandline. Every option I've tried so far with 7.15, however, yields a larger archive file than I got with 1.10. Does anyone have any suggestions?

11. Originally Posted by rpresser
My goal in this code-golf challenge is to produce the smallest possible compressed file plus commandline. Every option I've tried so far with 7.15, however, yields a larger archive file than I got with 1.10. Does anyone have any suggestions?
I've got a theory. A couple of years ago I posted that zpaq's file extension sorting can seriously hurt compression on certain types of content. In particular, huge source code trees such as the Linux kernel would benefit from keeping all files from the same directory closer together in the compression stream. With file sorting in zpaq, it'll break up the directory structure, which seems to hurt compression. A lot.

Please see my previous posts, where the Linux kernel was compressed between 5%–13% worse with the default file extension sorting in zpaq:

Originally Posted by traal
Almost two years later, and the ZPAQ file sorting still really hurts compression on source code file trees with many files. I re-did my tests on the current Linux kernel source code.

My tests show between 5% and 7% worse compression when the default file sorting is enabled, compared to the alphabetical order from the file system.
For this reason, I suggested a trivial patch to add a "-nosort" command line option. Sadly, my patch was never accepted. It still works, though, and my patch still improves compression for the Linux kernel source code and other similar projects.

This is an idea that Matt had back in 2014, and made the default, and as I discovered it unfortunately does not work everywhere. It can improve certain use cases, and make others significantly worse.

Originally Posted by Matt Mahoney
zpaq 6.44 sorts files by extension and then by full path before packing into blocks for compression.
//traal

12. I had same problem with 7z and tried making optimal sort order.
you can always sort them yourself with tar first.

13. Zpaq is used as a Kolmogorov complexity approximation tool in the following article:

Phylogeny of the COVID-19 Virus SARS-CoV-2 by Compression

The latter (=Kolmogorov complexity) being incomputable we approximate it by a good compressor such as the modern zpaq.
We provide a new alignment-free method based on compression to determine the phylogeny and taxonomy of the SARS-CoV-2 virus (the virus causing the COVID-19 pandemic). The method is based on compression, remarkably simple, very fast, and quantifies a distance to a number between 0 (identical) and 1 (totally different).
Data compression is used to analyze SARS-CoV-2. Who would have thought?

14. Originally Posted by Gotty
Who would have thought?
Surprising!

https://www.biorxiv.org/content/10.1...242v2.full.pdf

15. I need help changing the behavior of the list () function with the -all switch,
for two main problems.

(1) the "purge" of :

This is an example (Windows) of a folder named
z:\myzarc compressed to 1.zaq

Listing (l) call the list() function (line 3525 of 715's zpaq.cpp)

zpaq v7.15 journaling archiver, compiled Aug 17 2016
z:1.zpaq: 319 versions, 2388 files, 4081 fragments, 119.742900 MB

- 2020-02-18 00:00:54 165774017 DA z:/myzarc/
- 2015-09-05 13:48:20 0 A z:/myzarc/1
- 2015-09-07 14:38:10 367 A z:/myzarc/1.bat
- 2018-11-06 17:49:56 341409 A z:/myzarc/1.eml
- 2016-05-17 14:55:48 27648 A z:/myzarc/1.xls
- 2015-09-05 13:48:20 0 A z:/myzarc/1000
- 2015-09-05 13:48:22 0 A z:/myzarc/110

Using "-all" the : into the Windows path (z:\myzarc) is replaced by / (=>z/myzarc)

zpaq v7.15 journaling archiver, compiled Aug 17 2016
z:1.zpaq: 319 versions, 2388 files, 4081 fragments, 119.742900 MB

- 2019-02-25 11:34:18 152971946 0001/ +933 -0 -> 95751539
- 2019-02-25 11:34:18 152971946 DA 0001/z/myzarc/
- 2015-09-05 13:48:20 0 A 0001/z/myzarc/1
- 2015-09-07 14:38:10 367 A 0001/z/myzarc/1.bat <<<======== here's the problem, why z/myzarc and not z:/myzarc
- 2018-11-06 17:49:56 341409 A 0001/z/myzarc/1.eml
- 2016-05-17 14:55:48 27648 A 0001/z/myzarc/1.xls
- 2015-09-05 13:48:20 0 A 0001/z/myzarc/1000
0.062 seconds (all OK)

Can someone help me to patch the list() function to write out

It's not very clear (at least for me) how the
"printUTF8(p->first.c_str());" works in this case,
and the list() function is become a bit... messy over time

(2) where is the format of the versions?
"0001/z/myzarc/1.bat" seems like something of "printf("%4d") ... but... I cannot find how it works

Thanks

16. Originally Posted by Xi2
I wonder if there some other illumos/Solaris ZPAQ users
... in effect... yes

17. Originally Posted by xpk
Easier navigation between different versions. Timestamps for all.

Hello,
I have trouble navigating by using "version" indexes only. Would it be possible to assign short version comments or names?

Then there are the dates. I am not interested in archiving dates. I may re-archive the files later and the archiving dates would not represent the historical dates of the files. If I want to set version dates when the files were backed up via "-until", then ZPAQ shoots at me with "zpaq error: cannot truncate with an index". I obviously don't want to truncate anything, much less so while creating new archive. I don't want to mess with system time for each version addition either. I only want to set the timestamp. I also want to use multi-part archives with an index. So many great features. But the "-until" switch looks overused for many different things.

docs regarding current state:

ideal example:
Code:
zpaq add BigArchive???.zpaq c:\Largefileset\* -m4 -t4 -f -index BigArchive000.zpaq -until 2003-01-02 -comment version-1.0
Thanks.
WOW, 2018!

The short answer is: no, without breaking with older version

18. I enclose "my" version of zpaq (patch 24), not finished yet.

Added a series of switches for various real situations (use of zfs filesystem, backup of QNAP NAS etc).

Some improvement in the data shown during operations.

Added s ("size") command which calculates the total size of files.
Something like dir / s.
Obviously I use it on UNIX systems, where there is no easy way to calculate this size.

The most important change is missing, to make my GUI work faster on the time machine, but it is not easy to understand what happens "behind the scenes"

I also put a beta version of PAKKA (Win32 executable) that I am updating.
Essentially it is used to easily extract files, and show a sort of "time machine" of versions
in a zpaq archive (even multipart).

It is compiled at 32bit so it does not work for very large file lists (more than 4/5 million files),
work very slow (need to reverse engineering zpaq's output) etc. etc.

If I can edit zpaq list -all I will bring to 64bit and working version.

It's a developer build, with debug code, half italian half english, memory leaks and so on.
More than 2MB just for an "head.exe"

19. ## Thanks:

karlgustavv (7th September 2020)

20. The real question is... WHY this ?

// return a/b such that there is exactly one "/" in between, and
// in Windows, any drive letter in b the : is removed and there
// is a "/" after.
string append_path(string a, string b) {
int na=a.size();
int nb=b.size();
#ifndef unix
if (nb>1 && b[1]==':') { // remove : from drive letter
if (nb>2 && b[2]!='/') b[1]='/';
else b=b[0]+b.substr(2), --nb;
}
#endif
if (nb>0 && b[0]=='/') b=b.substr(1);
if (na>0 && a[na-1]=='/') a=a.substr(0, na-1);
return a+"/"+b;
}

21. PAKKA.EXE with 64bit custom sorter, some fix to unpack linux/*nix files and Windows one.
Does not work on 32bit (like Windows XP). In the future yes (...because I like it!)

22. This development release (with MPRESS) has improved file handling in Windows and * nix formats.
It is still full of debug code, memory leaks, partial internalization (some Italian text here and there)
etc

Essentially it is a windows graphical interface to extract files from single or multiple (indexed) ZPAQ archives, with the possibility of viewing the various versions of the files.

A tool specifically for programmers (restore previous versions of sources, with SEARCH!!!!!), dba (db dumps) etc.

It can (also) function as a kind of "time machine" (or PC-BSD / TrueOS time slider).

The program is a 32bit executable with a number of 32 and 64 bit programs linked inside which are used depending on the bitness

Two builds of zpaq customized by me with other functions are included (you can find them out of curiosity in the temp folder).

Any feedback and reporting will be highly appreciated

Better yet a mechanism to get an ordered list like in version 6.5!

23. WOW: zero feedback

This version starts to allow the creation of ZPAQ files, in debug version (it shows exactly what it does behind the scenes).

Anyway, given the lack of interest, I think it will be the last post on this topic, I will finish the development myself

24. I can confirm usage on Win XP SP3 Czech x32. I would recommend to polish some text (better English) and to improve design/visign - better/smaler font, to round some values etc.

Best regards,

FatBit

25. I was looking forward to a Total Commander plugin with improved versioning handling, similar to your program, unfortunately, it seems abandoned.

26. Originally Posted by fcorbelli
I enclose "my" version of zpaq (patch 24), not finished yet.

Added a series of switches for various real situations (use of zfs filesystem, backup of QNAP NAS etc).

Some improvement in the data shown during operations.

Added s ("size") command which calculates the total size of files.
Something like dir / s.
Obviously I use it on UNIX systems, where there is no easy way to calculate this size.

​Thank you for sharing your improvements. Especially "-nowindows" is appreciated.

Could you perhaps add the option that zpaq splits the archives after x MByte?

27. Originally Posted by FatBit
I can confirm usage on Win XP SP3 Czech x32. I would recommend to polish some text (better English) and to improve design/visign - better/smaler font, to round some values etc.

Best regards,

FatBit
It's just an alpha release,
In fact a delphi form of a much bigger software, 100%
Italian.
My english is just slightly better than my russian

28. This version, still beta, (maybe!) extracts files with UTF-8 name format (for debugging purposes enumerates all UTF-8 files into the log).

It is by no means trivial to create Windows batch files to support it, I had to write a small C program to change the code page (the chcp command does not work in batch).

I also had to change, again, the rename function of zpaq.cpp which, inexplicably (for me), changes the names of the files displayed by the l command with the -all option in a very strange way (?)

from

string append_path(string a, string b) {
int na=a.size();
int nb=b.size();
#ifndef unix
if (nb>1 && b[1]==':') {  // remove : from drive letter
if (nb>2 && b[2]!='/') b[1]='/';
else b=b[0]+b.substr(2), --nb;
}
#endif
if (nb>0 && b[0]=='/') b=b.substr(1);
if (na>0 && a[na-1]=='/') a=a.substr(0, na-1);
return a+"/"+b;
}

to this

string myappend_path(string a, string b) {
///  int na=a.size();
///  int nb=b.size();
/// if (nb>0 && b[0]=='/') b=b.substr(1);
///if (na>0 && a[na-1]=='/') a=a.substr(0, na-1);
return a+"|"+b;
}

Please some volunteers can try to extract files from both Windows (with or without letter)
c:\foo\bar
foo\bar,
qualità\pippo
Linux and BSD?

And even for multivolume, like
test_0001.zpaq
test_0002.zpaq
test_0003.zpaq
?

Thanks (aesthetic aspects and memory leaks are irrelevant, I'll fix them last).

PS if there is an old version inside% temp%\pkz I suggest deleting its contents,
need at least -franz28+ to work
zpaq v7.15-franz28 journaling archiver, compiled Sep 13 2020

29. Originally Posted by fcorbelli
This version, still beta, (maybe!) extracts files with UTF-8 name format (for debugging purposes enumerates all UTF-8 files into the log).

It is by no means trivial to create Windows batch files to support it, I had to write a small C program to change the code page (the chcp command does not work in batch).

I also had to change, again, the rename function of zpaq.cpp which, inexplicably (for me), changes the names of the files displayed by the l command with the -all option in a very strange way (?)

from

string append_path(string a, string b) {
int na=a.size();
int nb=b.size();
#ifndef unix
if (nb>1 && b[1]==':') {  // remove : from drive letter
if (nb>2 && b[2]!='/') b[1]='/';
else b=b[0]+b.substr(2), --nb;
}
#endif
if (nb>0 && b[0]=='/') b=b.substr(1);
if (na>0 && a[na-1]=='/') a=a.substr(0, na-1);
return a+"/"+b;
}

to this

string myappend_path(string a, string b) {
///  int na=a.size();
///  int nb=b.size();
/// if (nb>0 && b[0]=='/') b=b.substr(1);
///if (na>0 && a[na-1]=='/') a=a.substr(0, na-1);
return a+"|"+b;
}

Please some volunteers can try to extract files from both Windows (with or without letter)
c:\foo\bar
foo\bar,
qualità\pippo
Linux and BSD?

And even for multivolume, like
test_0001.zpaq
test_0002.zpaq
test_0003.zpaq
?

Thanks (aesthetic aspects and memory leaks are irrelevant, I'll fix them last).

PS if there is an old version inside% temp%\pkz I suggest deleting its contents,
need at least -franz28+ to work
zpaq v7.15-franz28 journaling archiver, compiled Sep 13 2020
Don't you have a github or public repository to make it easier to develop and update the application a bit outdated?, since the GPU could also be used for the necessary calculations for file compression
also update the encryption as the current one is outdated

30. Originally Posted by carlosnewmusic
Don't you have a github or public repository to make it easier to develop and update the application a bit outdated?, since the GPU could also be used for the necessary calculations for file compression
also update the encryption as the current one is outdated
I am not the developer of ZPAQ, but Matt Mahoney.
I do not know if GPU can be used, ZPAQ use a rather "uniqe" approach (maybe even "really weird")

Why do you think that encryption is outdated?

PS on
http://www.francocorbelli.it/pakka.exe
always the last release of my little software

31. 24.3 Some improvements, vaguely starting to seems like a pre release

Please any feedback on unpacking windows, linux and bsd files, possibly with utf-8 characters?
Thank you

Page 85 of 86 First ... 357583848586 Last