Page 85 of 85 FirstFirst ... 3575838485
Results 2,521 to 2,532 of 2532

Thread: zpaq updates

  1. #2521
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    215
    Thanks
    0
    Thanked 0 Times in 0 Posts
    So, I want to create an incremental archive like so
    Code:
    zpaq a backup????.zpaq "files" -index backup0000.zpaq
    But I have the limitation of 4GB per file (FAT32). So I split the large first archive
    Code:
    splits backup0001.zpaq 4000000
    And then rename the chunks to backup0002, 0003 etc.

    And so, the archive breaks! Can't extract anything.

    I tried to re-index it, but to no avail.

    How do I create an indexed incremental archive with a file size limitation?


    Thanks in advance!

  2. #2522
    Member
    Join Date
    Jun 2015
    Location
    Moscow
    Posts
    12
    Thanks
    4
    Thanked 2 Times in 2 Posts
    Quote Originally Posted by nanoflooder View Post
    So, I want to create an incremental archive like so
    Code:
    zpaq a backup????.zpaq "files" -index backup0000.zpaq
    But I have the limitation of 4GB per file (FAT32). So I split the large first archive
    Code:
    splits backup0001.zpaq 4000000
    And then rename the chunks to backup0002, 0003 etc.

    And so, the archive breaks! Can't extract anything.

    I tried to re-index it, but to no avail.

    How do I create an indexed incremental archive with a file size limitation?


    Thanks in advance!
    Zpaq does not have flags that allow you to limit the size of the archive

  3. #2523
    Member
    Join Date
    Jan 2018
    Location
    spain
    Posts
    5
    Thanks
    3
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by SpyFX View Post
    Zpaq does not have flags that allow you to limit the size of the archive
    There should be to be able to limit the file size

  4. #2524
    Member
    Join Date
    May 2019
    Location
    Rome
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts
    I have a big problem with ZPAQ 7.15 : it segfaults when compiled into linux systems that use MUSL library instead of standard libc (Alpine Linux in my case).

    The problem seems related to the stack size of the library, that is limited to 80kb instead of the 8Mb (!) provided by gLibc, causing overflows during pthread-s:

    https://github.com/richfelker/musl-cross-make/issues/33

    Is there a workaround for the issue ?

  5. #2525
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,943
    Thanks
    291
    Thanked 1,286 Times in 728 Posts
    Maybe compile with non-default stack size via gcc -Wl,--stack,16777216 ?
    http://www.mattmahoney.net/dc/zpaq715.zip

  6. #2526
    Member
    Join Date
    May 2019
    Location
    Rome
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Shelwien View Post
    Maybe compile with non-default stack size via gcc -Wl,--stack,16777216 ?
    http://www.mattmahoney.net/dc/zpaq715.zip
    Unfortunately, this is not working.

    Even if compiled -static, it segfaults almost immediately.

    I'm a Sys-op guy, not really into compiling and coding, this is what I get running into gdb:

    Starting program: /root/zpaq a test /
    zpaq v7.15 journaling archiver, compiled May 3 2019
    test.zpaq: 0 versions, 0 files, 0 fragments, 0.000000 MB
    Updating test.zpaq at offset 0 + 0
    Adding 1326.112348 MB in 42966 files -method 14 -threads 1 at 2019-05-03 12:12:34.
    [New LWP 2317]
    [New LWP 2318]
    1.27% 0:00:27 [1..236] 16754818 -method 14,97,0

    Thread 2 "zpaq" received signal SIGSEGV, Segmentation fault.
    [Switching to LWP 2317]
    0x00007ffff7fac5c0 in libzpaq::Predictor::Predictor(libzpaq::ZPAQL&) ()
    (gdb)


    (gdb) backtrace
    #0 0x00007ffff7fac5c0 in libzpaq::Predictor::Predictor(libzpaq::ZPAQL&) ()
    #1 0x00007ffff7fc7122 in libzpaq::compressBlock(libzpaq::StringBuffer*, libzpaq::Writer*, char const*, char const*, char const*, bool) ()
    #2 0x00007ffff7f83284 in compressThread(void*) ()
    #3 0x00007ffff7fdd2c6 in start (p=<optimized out>) at src/thread/pthread_create.c:147
    #4 0x00007ffff7fddd11 in __clone () at src/thread/x86_64/clone.s:21
    Backtrace stopped: frame did not save the PC
    (gdb)

    (gdb) frame 3
    #3 0x00007ffff7fdd2c6 in start (p=<optimized out>) at src/thread/pthread_create.c:147
    147 src/thread/pthread_create.c: No such file or directory.

  7. #2527
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,943
    Thanks
    291
    Thanked 1,286 Times in 728 Posts
    Try compiling with -DNOJIT ?

  8. #2528
    Member
    Join Date
    May 2019
    Location
    Rome
    Posts
    3
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Shelwien View Post
    Try compiling with -DNOJIT ?
    Still no luck on this. I think I'll give up on this Alpline Linux and try something that has glibc instead of musl.

  9. #2529
    Member
    Join Date
    Mar 2020
    Location
    Northern Hemisphere
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hello, I have a question about zpaq. I have a use case where different people must add directories with the same name, but differing contents to the same archive. All files that get added are necessary (we are basically deleting nothing). zpaq would be ideal for this because of the great deduplication and compression.

    When different people add directories with the same name but different files, a normal extract only gives the latest set of added files because zpaq thinks the rest got deleted from the directory. That leads to my question: is there a way to use zpaq in an append-only mode (journaling which records no delete actions) which still has compression and deduplication? I know I can just use -all switch and deal with the files myself, but I was hoping for an easier way.

    I also want to thank you for the great tool. I use it daily, mainly because of the deduplication. Sorry for any mistakes, I'm not a native speaker.

  10. #2530
    Member
    Join Date
    Jun 2020
    Location
    New Jersey
    Posts
    1
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Today as an entry to this Code Golf challenge I started playing with zpaq. When I used apt-get install zpaq on an Ubuntu 18.04 virtual machine, I got zpaq version 1.10, and produced a fairly good result. Then someone pointed out that 1.10 is over ten years old, and the current version of zpaq is 7.15.

    It seems that the commandline interface underwent a major change in version 2, which isn't surprising. But what is surprising:
    * a few of the options from 1.10 relating to metadata do not have any equivalent win 7.15: n (don't store filenames), s (don't store SHA1 checksums), i (don't store file sizes), q (no messages)
    * Even skipping the metadata problem, I'm unable to figure out what current -method options to use to reproduce the output from 1.10.

    My goal in this code-golf challenge is to produce the smallest possible compressed file plus commandline. Every option I've tried so far with 7.15, however, yields a larger archive file than I got with 1.10. Does anyone have any suggestions?

  11. #2531
    Member
    Join Date
    Feb 2015
    Location
    Sweden
    Posts
    3
    Thanks
    0
    Thanked 2 Times in 2 Posts
    Quote Originally Posted by rpresser View Post
    My goal in this code-golf challenge is to produce the smallest possible compressed file plus commandline. Every option I've tried so far with 7.15, however, yields a larger archive file than I got with 1.10. Does anyone have any suggestions?
    I've got a theory. A couple of years ago I posted that zpaq's file extension sorting can seriously hurt compression on certain types of content. In particular, huge source code trees such as the Linux kernel would benefit from keeping all files from the same directory closer together in the compression stream. With file sorting in zpaq, it'll break up the directory structure, which seems to hurt compression. A lot.

    Please see my previous posts, where the Linux kernel was compressed between 5%–13% worse with the default file extension sorting in zpaq:

    Quote Originally Posted by traal View Post
    Almost two years later, and the ZPAQ file sorting still really hurts compression on source code file trees with many files. I re-did my tests on the current Linux kernel source code.

    My tests show between 5% and 7% worse compression when the default file sorting is enabled, compared to the alphabetical order from the file system.
    For this reason, I suggested a trivial patch to add a "-nosort" command line option. Sadly, my patch was never accepted. It still works, though, and my patch still improves compression for the Linux kernel source code and other similar projects.

    This is an idea that Matt had back in 2014, and made the default, and as I discovered it unfortunately does not work everywhere. It can improve certain use cases, and make others significantly worse.

    Quote Originally Posted by Matt Mahoney View Post
    zpaq 6.44 sorts files by extension and then by full path before packing into blocks for compression.
    //traal
    Last edited by traal; 22nd June 2020 at 12:08. Reason: Added some info

  12. #2532
    Member
    Join Date
    Jun 2018
    Location
    Yugoslavia
    Posts
    58
    Thanks
    8
    Thanked 3 Times in 3 Posts
    I had same problem with 7z and tried making optimal sort order.
    you can always sort them yourself with tar first.

Page 85 of 85 FirstFirst ... 3575838485

Similar Threads

  1. ZPAQ self extracting archives
    By Matt Mahoney in forum Data Compression
    Replies: 31
    Last Post: 17th April 2014, 03:39
  2. ZPAQ 1.05 preview
    By Matt Mahoney in forum Data Compression
    Replies: 11
    Last Post: 30th September 2009, 04:26
  3. zpaq 1.02 update
    By Matt Mahoney in forum Data Compression
    Replies: 11
    Last Post: 10th July 2009, 00:55
  4. Metacompressor.com benchmark updates
    By Sportman in forum Data Compression
    Replies: 79
    Last Post: 22nd April 2009, 03:24
  5. ZPAQ pre-release
    By Matt Mahoney in forum Data Compression
    Replies: 54
    Last Post: 23rd March 2009, 02:17

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •