zpaq updates

[Xi2]

Solaris - iLLumos - SmartOS

Hi, I'm wondering if binaries of ZPAQ are a available for illumos/Solaris operating systems, I plan to use ZPAQ in combination with ZFS and much rather use it on an illumos based operating system than in Linux, any help would be greatly appreciated.

[Gonzalo]

Solaris - iLLumos - SmartOS

Hi, I'm wondering if binaries of ZPAQ are a available for illumos/Solaris operating systems, I plan to use ZPAQ in combination with ZFS and much rather use it on an illumos based operating system than in Linux, any help would be greatly appreciated.

That's interesting... But first things first. I'm sorry I cannot help you with your binaries. Unless I do some cross-compiling. I'll check later if it is possible.

Anyway... I think that ZFS+ZPAQ is kind of redundant. How exactly are you planning on put these two together?

[Xi2]

That's interesting... But first things first. I'm sorry I cannot help you with your binaries. Unless I do some cross-compiling. I'll check later if it is possible.

Anyway... I think that ZFS+ZPAQ is pretty redundant. How exactly are you planning on put these two together?

I don't intent to use deduplication in ZFS, I have a SmartOS Zone that serves files on a network of 15 computers, for backups I use Incremental ZFS Send and as additional backup I intent to use ZPAQ archives replicated with RClone to something like B2 BackBlaze

[Xi2]

I wonder if there some other illumos/Solaris ZPAQ users

[Matt Mahoney]

Only source code and Windows executables. There are makefiles for Linux.

[Gonzalo]

There is a halted project intended to provide binary compatibility:

https://github.com/keremet/Illumos--...without-zones-

I know is a long shot but maybe you are proficient enough to complete it or at least adapt it to your needs.


See also this:


https://docs.oracle.com/cd/E19044-01...pea/index.html

The lx brand uses the branded zones framework to enable Linux binary applications to run unmodified on a machine with a Solaris Operating System kernel.

[Xi2]

Yes @Gonzalo you are correct, under SmartOS you can run unmodified Linux binaries. (On a LX Branded Zone)

[Xi2]

I'm testing ZPAQ with a data sample under Windows 2008 64bits

After several runs I wanted to test extraction but encountered this error, any clues as to what's happening? thx.

F:\Zpaq1>zpaq extract backup???? -all -test
zpaq v7.15 journaling archiver, compiled Aug 17 2016
backup????.zpaq: Unordered fragment tables: expected >= 879172 found 877358
25 versions, 211368 files, 879222 fragments, 46210.832505 MB
zpaq error: unordered frags
2.932 seconds (with errors)


F:\Zpaq1>zpaq extract backup????
zpaq v7.15 journaling archiver, compiled Aug 17 2016
backup????.zpaq: Unordered fragment tables: expected >= 879172 found 877358
25 versions, 211368 files, 879222 fragments, 46210.832505 MB
zpaq error: unordered frags
2.262 seconds (with errors)

[Xi2]

Could this be data corruption?
I'm able to list the archive however extracting is not working

[pswhar]

Did you try sequentially to check which version is corrupt?
F:\Zpaq1>zpaq extract backup???? -until [1..25] -test

[Xi2]

Did you try sequentially to check which version is corrupt?
F:\Zpaq1>zpaq extract backup???? -until [1..25] -test

Thank you for replying, the archive has a total of 25 versions and the error starts at version 24

What can be done in cases like this? What could be the cause of the error?

[SpyFX]

Thank you for replying, the archive has a total of 25 versions and the error starts at version 24

What can be done in cases like this? What could be the cause of the error?

to delete 24-25 files....

[Xi2]

to delete 24-25 files....

Funny I also went to GitHub to look at the source code since the error appears to be very uncommon...

Is there a way to delete only specific versions? or it is better to extract and re-ZPAQ the un-corrupted versions?

[SpyFX]

Is there a way to delete only specific versions?

no...

or it is better to extract and re-ZPAQ the un-corrupted versions?

how? extract =>> backup????.zpaq: Unordered fragment tables: expected >= 879172 found 877358


to delete 24 and all after it... next zpaq add ... create file with version 24....

[pswhar]

As SpyFx suggests:
if you delete file backup024 and backup025, next time executing
zpaq a backup???? [yourfiles]
will correct the problem...

i am not quite sure, but you could also execute
zpaq a backup???? [yourfiles] -until 24

This should truncate version 24 and append present state of your files...

[A:C]

Hi there, I'm new here. And of cause I have a question.

I am using zpaq 7.15 as journaling backup. If I extract with "-until -1 -f" to get older version over existing files I have a problem.
If newer file is bigger in size and I extract the older smaller one over it the resulting size is still the bigger one and fragments of (unwanted) newer file stays at end of recovered file.
If I delete the unwanted file before everything is fine. But I think recovering to an earlier state of backup is a feature of zpaq.
What I am doing wrong?
I am using Windows7 and standard NTFS.

Of cause I can delete the whole path before but than recovering takes very long time (large project).

regards

[Xi2]

bad frag IDs, skipping Messages...


Hi, I'm testing a section of a multi-part archive and encoutered this message


$DATA: bad frag IDs, skipping...


What does it mean? is the integrity of the archive compromised?


Thx.






C:\Users\Admin>zpaq64 x \\192.168.1.50\e$\data\backup\NetBack00005.zpaq -test
zpaq v7.15 journaling archiver, compiled Aug 17 2016
//192.168.1.50/e$/data/backup/NetBack00005.zpaq: 1 versions, 223 files, 2437324 fragments, 0.145247 MB
User/Downloads/ntfslinksview-x64.zip:Zone.Identifier:$DATA: bad frag IDs, skipping...
User/Downloads/ntfslinksview-x64/NTFSLinksView.chm:Zone.Identifier:$DATA: bad frag IDs, skipping...
User/Downloads/ntfslinksview-x64/readme.txt:Zone.Identifier:$DATA: bad frag IDs, skipping...
Extracting 0.198122 MB in 9 files -threads 4
[2437317..2437324] -> 198165
> User/Downloads/ntfslinksview-x64.zip
> User/Downloads/ntfslinksview-x64/NTFSLinksView.cfg
> User/Downloads/ntfslinksview-x64/NTFSLinksView.chm
> User/Downloads/ntfslinksview-x64/NTFSLinksView.exe
> User/Downloads/ntfslinksview-x64/readme.txt
0.421 seconds (all OK)

[carlosnewmusic]

GPU could be supported for faster processing, since GPUs now process certain operations faster than CPUs
https://developer.nvidia.com/computeworks

in the sense that GPU cores were added to existing CPUs for processing, and in case of failure when starting processing with GPU + CPU, only use the CPU with the existing cores :/

[Gonzalo]

Bug report.

Zpaq exits with no error whatsoever when disk is full and is not possible to write anything more.

zpaq x Androids.zpaq -repack Androids_.zpaq password
zpaq v7.15 journaling archiver, compiled Oct 30 2017
Androids.zpaq: 3 versions, 40253 files, 250479 fragments, 15333.292342 MB
Data 12687005723 -> 26651
Androids.zpaq 15333292342 -> Androids_.zpaq 61765
225.532 seconds (all OK)

[lSOg16gqyh]

Hi!

I do not know whether "decremental" backups are supported but I would encourage and welcome it. It means, roughly: among versions, the last is stored "as whole" (no-diff), and previous versions as diffs (relative to the last). When adding a "new last": store the new last as whole, and transform the so-far-last to a diff.

The extractor bytecode is always embedded? Even when it is a very common ("de facto standard") byte sequence? What about only referencing it?

When adding a (new) version to an archive: repacking (of the so-far content) is never carried out? Isn't there such thins as learning? With the new knowledge coming from the added content, the old content could be compressed better.. couldn't?

[Bulat Ziganshin]

"decremental" backups aren't that hard to implement, but it will mean that each backup operation will spend as much time as full backup, plus require double hdd space if we don't want to delete recent backup before we finished the current one

>With the new knowledge coming from the added content, the old content could be compressed better.. couldn't?

we can either compress new content based on old one or opposite

[FatBit]

Dear colleagues,

I have made quick zpaq x32 ver. 7.15 compression test - almost same text logs (80 files, ~ 1,3 GB). Results (sizing, timing) in picture. Test script in attachment.

But there is something wrong in script/system. If source is "enough big (like 49 GB, 28 kfs)" then script part (Dir %Target%%%i%%j.zpaq >> %Target%SizeLog.asc) && (Del %Target%%%i%%j.zpaq) will stop working and drive will fill with files.

Best regards,

FatBit

[xpk]

Easier navigation between different versions. Timestamps for all.

Hello,
I have trouble navigating by using "version" indexes only. Would it be possible to assign short version comments or names?


Then there are the dates. I am not interested in archiving dates. I may re-archive the files later and the archiving dates would not represent the historical dates of the files. If I want to set version dates when the files were backed up via "-until", then ZPAQ shoots at me with "zpaq error: cannot truncate with an index". I obviously don't want to truncate anything, much less so while creating new archive. I don't want to mess with system time for each version addition either. I only want to set the timestamp. I also want to use multi-part archives with an index. So many great features. But the "-until" switch looks overused for many different things.

docs regarding current state:

-until date | [-]version
When a date is specified, the update will be timestamped with date rather than the current date.
...
add -until is an error on multi-part archives or with an index.

ideal example:

zpaq add BigArchive???.zpaq c:\Largefileset\* -m4 -t4 -f -index BigArchive000.zpaq -until 2003-01-02 -comment version-1.0

Thanks.

[jimbow]

Hi,

Just tryed latest powerarchiver 2017, it cannot extract/test archives created with zpaq 7.15. Which version is used in PA? or any idea why it can't?
thanks

[pothos2]

zpaq 7.15 is now included in Debian sid: https://packages.debian.org/sid/zpaq

So sooner or later it will also be in the Debian-based distributions of the Microsoft Windows Subsystem for Linux which will give you a sane shell

(libzpaq is not a separate package and zpaqd is not packaged)

[carlosnewmusic]

zpaq 7.15 is now included in Debian sid: https://packages.debian.org/sid/zpaq

So sooner or later it will also be in the Debian-based distributions of the Microsoft Windows Subsystem for Linux which will give you a sane shell

(libzpaq is not a separate package and zpaqd is not packaged)

then with the debian version you can not apply the utilities(http://mattmahoney.net/dc/zpaqutil.html), right?

[pothos2]

Yes, zpaqd is not packaged (The cfg files could be included as examples) but it would be possible if libzpaq is also separated into an own package. You could ask the current maintainer through the bug tracker.

[MiroGeorg]

Guys, when ZPAQ is started from Task Scheduler under windows there is a problem with excessive swapping.
The problem is that Task Scheduler sets PagePriority to LOW, so ZPAQ's process memory is actively swapped to disk. The consequences of this behaviour is long, slow archiving progress with low CPU usage, and constant excessive hdd activity.

Problem can be partly solved with Process Hacker 2 ( rightclick on ZPAQ -> Miscellaneous -> Page Priority -> Normal )

or with command line:
ProcessHacker.exe -c -ctype process -cobject %pid% -caction priority -cvalue idle
ProcessHacker.exe -c -ctype process -cobject %pid% -caction pagepriority -cvalue 5


But the best, permanent solution is .REG:

Windows Registry Editor Version 5.00


[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\zpaq64_intel.exe]


[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\zpaq64_intel.exe\PerfOptions]
"CpuPriorityClass"=dword:00000001
"IoPriority"=dword:00000002
"PagePriority"=dword:00000005

you can replace zpaq64_intel.exe with zpaq64.exe or zpaq.exe

[MiroGeorg]

I ended up scheduling with https://www.z-cron.com/, because build in windows task scheduler is flawed by design.

Guys, when ZPAQ is started from Task Scheduler under windows there is a problem with excessive swapping.
The problem is that Task Scheduler sets PagePriority to LOW, so ZPAQ's process memory is actively swapped to disk. The consequences of this behaviour is long, slow archiving progress with low CPU usage, and constant excessive hdd activity.

Problem can be partly solved with Process Hacker 2 ( rightclick on ZPAQ -> Miscellaneous -> Page Priority -> Normal )

or with command line:
ProcessHacker.exe -c -ctype process -cobject %pid% -caction priority -cvalue idle
ProcessHacker.exe -c -ctype process -cobject %pid% -caction pagepriority -cvalue 5


But the best, permanent solution is .REG:

Windows Registry Editor Version 5.00


[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\zpaq64_intel.exe]


[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\zpaq64_intel.exe\PerfOptions]
"CpuPriorityClass"=dword:00000001
"IoPriority"=dword:00000002
"PagePriority"=dword:00000005

you can replace zpaq64_intel.exe with zpaq64.exe or zpaq.exe

[carlosnewmusic]

does not defragment,What can I do to recover my information?

it is a file encoded with -key