Page 10 of 14 FirstFirst ... 89101112 ... LastLast
Results 271 to 300 of 395

Thread: Zstandard

  1. #271
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    876
    Thanks
    80
    Thanked 315 Times in 219 Posts
    Quote Originally Posted by Shelwien View Post
    Input file:
    4,119,056,656 bytes - index file

    Output:
    bro 0 1,554,763,776 bytes 18.118 20.298
    bro 1 1,506,680,491 bytes 21.368 19.923
    bro 2 1,228,040,859 bytes 42.775 17.188
    bro 3 1,214,011,855 bytes 54.726 16.141
    bro 4 1,140,855,628 bytes 66.145 16.329
    bro 5 1,030,055,164 bytes 151.311 15.016
    bro 6 985,503,532 bytes 207.582 15.157
    bro 7 910,596,954 bytes 326.885 13.370
    bro 8 895,367,242 bytes 441.507 14.172
    bro 9 886,557,404 bytes 600.735 13.016
    bro 10 806,625,472 bytes 3311.095 14.297
    bro 11 785,969,012 bytes 8046.214 13.797

    zstd 1 1,404,187,617 bytes 17.036 3.828
    zstd 2 1,274,813,612 bytes 21.497 4.015
    zstd 3 1,176,226,579 bytes 28.134 4.484
    zstd 4 1,129,158,517 bytes 29.327 4.125
    zstd 5 1,117,910,735 bytes 40.646 4.578
    zstd 6 1,042,080,002 bytes 49.036 3.953
    zstd 7 1,019,551,302 bytes 56.867 4.390
    zstd 8 1,001,949,208 bytes 73.337 3.865
    zstd 9 985,327,675 bytes 93.599 4.312
    zstd 10 984,792,872 bytes 98.328 3.844
    zstd 11 945,979,707 bytes 145.297 4.344
    zstd 12 940,640,409 bytes 157.852 3.781
    zstd 13 930,793,717 bytes 226.151 4.281
    zstd 14 926,103,064 bytes 257.076 3.750
    zstd 15 922,318,814 bytes 412.030 4.203
    zstd 16 881,853,545 bytes 489.436 3.765
    zstd 17 865,358,829 bytes 632.937 4.297
    zstd 18 827,763,798 bytes 816.216 3.625
    zstd 19 812,424,273 bytes 1174.923 4.219

  2. Thanks:

    Cyan (4th December 2016)

  3. #272
    Member
    Join Date
    Jun 2010
    Location
    Portugal
    Posts
    22
    Thanks
    1
    Thanked 1 Time in 1 Post
    Quote Originally Posted by stbrumme View Post
    @Alvaro: If I remember correctly, the Syzygy chess endgame tablebases are compressed with LZ4.
    No, Syzygy TBs are compressed using a scheme from Alistair Moffat, a scheme I don't understand.
    Compression is a domain I don't understand, so I'm looking for an easy way out.
    But thanks anyway.

    Alvaro

  4. #273
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    876
    Thanks
    80
    Thanked 315 Times in 219 Posts
    Is it a bug or feature that zstd even levels decompress 10-11% faster then uneven levels?

  5. #274
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    869
    Thanks
    470
    Thanked 261 Times in 108 Posts
    > Is it a bug or feature that zstd even levels decompress 10-11% faster then uneven levels?

    It's likely a bug in the speed measurement routine.
    mem-to-mem benchmark don't show this pattern
    (you can use ./zstd -b to check).

  6. #275
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    869
    Thanks
    470
    Thanked 261 Times in 108 Posts
    There's a new release of Zstandard, v1.1.2 : https://github.com/facebook/zstd/releases/tag/v1.1.2

    Among the set of new features, there is a new build target prepared by @inikep, named gzstd, which is able to decompress both zstd and gzip files. gzstd depends on zlib dll, so it tests if the library is present on local system before building the executable. When it detects that zlib is not present, the executable only supports zstd files.

    We are looking for feedback on how gzstd builds. The long-term intention is that it could become default version, but we first need to ensure it builds correctly on most systems.

  7. Thanks (2):

    load (15th December 2016),willy (17th December 2016)

  8. #276
    Member
    Join Date
    Dec 2011
    Location
    Cambridge, UK
    Posts
    486
    Thanks
    168
    Thanked 166 Times in 114 Posts
    So is the plan here that libzstd.so can be linked against instead of libz.so, with a compatible API and header files so that decompression can read deflate while compression writes zstd, allowing for seamless migration, albeit in a one-way only direction? Do we lose much flexibility by interacting with libzstd using a zlib API? I assume things like the windowbits fields get ignored, or change meaning, but options like Z_SYNC_FULL and Z_SYNC_FLUSH can be handy for random-access compressed data streams.

    As it's dependant on a libz.so / zlib.dll I assume this means it works with any of the zlib implementations (eg zlib, zlib-ng, cloudflare and intel optimised zlibs)? They all ought to have the same API so I think it ought to work fine, unless you really are on some ancient DOS system with FAR pointers... *shudder*.

  9. #277
    Programmer
    Join Date
    May 2008
    Location
    PL
    Posts
    309
    Thanks
    68
    Thanked 173 Times in 64 Posts
    There are two independent things in zstd:
    a) a possibility of decompression of .gz files using zstd.exe (zstdcli.c and fileio.c) that uses zlib's inflate()
    b) zlibWrapper which allows to compile programs that use zlib API with zstd and switch compression between zlib and zstd

    The list of supported functions and benchmarks for zlibWrapper vs zlib are available at https://github.com/facebook/zstd/tree/dev/zlibWrapper
    We didn't try to compile it with other zlib implementations but it should work.

  10. Thanks:

    JamesB (15th December 2016)

  11. #278
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    204
    Thanks
    170
    Thanked 109 Times in 65 Posts
    if i decompress a .gz archive using zstd.exe under windows, i end up with a TAR file (with no extension).

    is that expected?

  12. #279
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    869
    Thanks
    470
    Thanked 261 Times in 108 Posts
    if i decompress a .gz archive using zstd.exe under windows, i end up with a TAR file (with no extension).
    is that expected?
    A *.tar.gz becomes a *.tar

    yes, this behaviour is supposed to be identical to gzip / gunzip.

  13. #280
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    204
    Thanks
    170
    Thanked 109 Times in 65 Posts
    sorry, i had to leave the house, and soon i realized this was a silly question.

  14. #281
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    204
    Thanks
    170
    Thanked 109 Times in 65 Posts
    someone made a plugin for total commander.

    http://www.ghisler.ch/board/viewtopi...969&highlight=

  15. Thanks:

    Cyan (24th December 2016)

  16. #282
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    869
    Thanks
    470
    Thanked 261 Times in 108 Posts
    There's a new release of Zstandard,
    v1.1.3 : https://github.com/facebook/zstd/releases/tag/v1.1.3

    One of its new features is experimental multi-threading support.
    Since it's experimental, it's not merged by default into "main" targets zstd and libzstd.
    Instead, it has its own target, `make zstdmt`.

    When using it, it's possible to select the number of threads, using `-T#`.
    For example : ./zstdmt -T2 file

    One of the most important properties we are trying to ensure is the portability of the MT-enabled code path.
    It was checked on several "common" systems, but there is always a chance something could go wrong on some untested system.
    So we are welcoming feedback on this topic.

  17. Thanks (4):

    encode (6th February 2017),inikep (6th February 2017),load (6th February 2017),milky (6th February 2017)

  18. #283
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    204
    Thanks
    170
    Thanked 109 Times in 65 Posts
    some windows .exe files.

    but i guess windows has been tested before...
    Attached Files Attached Files

  19. Thanks:

    Cyan (6th February 2017)

  20. #284
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    869
    Thanks
    470
    Thanked 261 Times in 108 Posts
    Thanks @load, yes it has,
    we are more worried of targets like Solaris, DragonFly, and other various unixes.

  21. #285
    Member
    Join Date
    May 2012
    Location
    Germany
    Posts
    39
    Thanks
    25
    Thanked 103 Times in 24 Posts
    Just for info

    - 7-Zip ZS Edition is updated: https://github.com/mcmilk/7-Zip-zstd/releases
    - lz4mt, lz5mt and zstdmt windows binaries also: https://github.com/mcmilk/zstdmt/releases

  22. Thanks (4):

    Bulat Ziganshin (9th February 2017),Cyan (7th February 2017),inikep (9th February 2017),load (7th February 2017)

  23. #286
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,532
    Thanks
    755
    Thanked 674 Times in 365 Posts
    can you plese briefly descriibe what is meant by m/t support? independent blocks compression, rar-like m/t matchfinder with the single stream, something in-between? and what are your future plans in m/t and gpu support in zstd?

  24. #287
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    869
    Thanks
    470
    Thanked 261 Times in 108 Posts
    m/t support in Zstandard produces compressed frames which are externally undistinguishable from single thread ones.
    While blocks are processed in parallel, they are not independent : each successive block borrow a part of previous one.

    Blocks can be made fully independent by using advanced (hidden) parameter `--zstd=overlapLog=0`.
    Blocks can be made fully overlapped by setting `--zstd=overlapLog=9`, which produces best compression but also slower speed.
    Default value is 6, except for `--ultra -22`, which uses `overlapLog=9`.
    (as a funny consequence, compressing with MT support at level 22 tends to produce slightly _better_ compression ratio than single thread, though you'll need a lot of data to witness any difference).

    Block sizes can also be set manually, using `-B#` command. For example `-B10MB` means "blocks of 10 MB".
    Default block size is 4x WindowSize (which varies depending on compression level).

    The medium term plan is to merge the MT API inside the regular API.
    That will require to implement a new stable mechanism to set advanced parameters, since nbThreads will become one of them.

  25. Thanks:

    Bulat Ziganshin (9th February 2017)

  26. #288
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,532
    Thanks
    755
    Thanked 674 Times in 365 Posts
    Quote Originally Posted by Cyan View Post
    One of the most important properties we are trying to ensure is the portability of the MT-enabled code path.
    It was checked on several "common" systems, but there is always a chance something could go wrong on some untested system.
    So we are welcoming feedback on this topic.
    my own experience suggests using extra API layer for everything that may be need to extended. for example, freearc has own classes for File and Archive, and that simplified adding new functionality unavailable with standard Haskell file API

    here, i propose to add zstd own Threading API layer. Since pthreads, C++ 11 threads and C 11 threads have essentially the same API, i propose just to add its version with zstd_ prefix and try to steal WinAPI implementation, f.e. here

    so, at the end of day, you will have 100% compatibility with modern C/C++ compilers as well as support for older compilers on most modern platforms

    alternatively, you may use TinyThread as is and contribute C/C++ 11 support directly to that lib

  27. Thanks:

    Cyan (9th February 2017)

  28. #289
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,532
    Thanks
    755
    Thanked 674 Times in 365 Posts
    i think i've found a subtle bug. here we use previous data as initial dictionary. but ZSTD_compressBegin_advanced calls ZSTD_compress_insertDictionary that treats first 4 bytes as magic number, so these raw data may be interpreted as special cdict

    the same applies to ZSTD_compressBegin_usingDict - it also calls ZSTD_compress_insertDictionary, and i bet it's error since there is also ZSTD_compressBegin_usingCDict

    overall, you need to 100% distinguish between dict (raw data) and cdict (special format). may be, if cdict contains raw data, they should be prefixed too? and each API function should manifest whether it works with dict or cdict (f.e. using dict/cdict as parameter name), so you and your users can't use it wrong

  29. Thanks:

    Cyan (9th February 2017)

  30. #290
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    869
    Thanks
    470
    Thanked 261 Times in 108 Posts
    Thanks @Bulat, I believe you are right.
    The probability for this scenario to happen is low, but still, better be safe than sorry.
    I need to add a method which guarantees that dictionary will be loaded as "rawData" only, with no chance for it to be confused with a prepared zstd dictionary.

  31. #291
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,532
    Thanks
    755
    Thanked 674 Times in 365 Posts
    yeah, and expose internal functions you have used to create mtzstd, as well as ZSTD_loadDictionaryContent. I had idea to write m/t engine like your one, but turned down by lack of this function (and lack of time but that doesn't counts ). At least this helped to find the bug, since i read the source exactly to see how you use zstd API to fill dictionary without compressing these data.

  32. #292
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,532
    Thanks
    755
    Thanked 674 Times in 365 Posts
    https://github.com/facebook/zstd/releases/tag/v1.2.0

    Major features :
    • Multithreading is enabled by default in the cli. Use -T# to select nb of thread. To disable multithreading, build target zstd-nomt or compile with HAVE_THREAD=0.
    • New dictionary builder named "cover" with improved quality (produces better compression ratio), by @terrelln. Legacy dictionary builder remains available, using --train-legacy command.

  33. Thanks (4):

    comp1 (12th May 2017),Cyan (12th May 2017),Simorq (21st May 2017),SolidComp (13th May 2017)

  34. #293
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    735
    Thanks
    230
    Thanked 231 Times in 142 Posts
    "Zstandard, or "zstd" (pronounced "zee standard"), is a data compression mechanism. This document describes the mechanism, and registers a media type to be used when transporting zstd-compressed via Multipurpose Internet Mail Extensions (MIME).": https://tools.ietf.org/id/draft-kuch...h-zstd-00.html

  35. Thanks:

    milky (28th September 2017)

  36. #294
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    869
    Thanks
    470
    Thanked 261 Times in 108 Posts
    Yes, it's basically the Zstandard format specification redrafted to fit IETF requirements for an RFC document.
    We'll see how it goes. I'm told it can be a long process.

  37. Thanks (2):

    Jarek (27th September 2017),milky (28th September 2017)

  38. #295
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    735
    Thanks
    230
    Thanked 231 Times in 142 Posts

  39. #296
    Member
    Join Date
    May 2014
    Location
    Canada
    Posts
    137
    Thanks
    62
    Thanked 21 Times in 12 Posts
    Quote Originally Posted by Jarek View Post
    Is it correct that zstd is tuned more for text than binary?

  40. #297
    Member
    Join Date
    May 2014
    Location
    Canada
    Posts
    137
    Thanks
    62
    Thanked 21 Times in 12 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    ...and what are your future plans in m/t and gpu support in zstd?
    Yes, also very interested in the GPU plans. I would be interested in helping with this, if help is needed.

  41. #298
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    869
    Thanks
    470
    Thanked 261 Times in 108 Posts
    There is a new release of Zstandard, featuring a "long range mode", developed by our intern Stella Lau :
    https://github.com/facebook/zstd/releases/tag/v1.3.2
    It significantly improves compression ratio of large files/archives when similar segments can be found at large distance.
    The release also includes pre-compiled Windows binaries.

    The release note includes a feature description, as well as its performance, compared to lrzip, another compressor based on similar strategy.
    Readers of this forum will also recognise Bulat's srep preprocessor, which is in the same category.

    The biggest difference is that LRM still produces "normal" Zstandard frames, hence compression and decompression remain "single pass".
    For decompression speed, it makes a noticeable difference.

    For this feature, maximum window size has been upgraded to 2 GB (users must select it explicitly, using `--long=31`, otherwise it keeps default 128MB limit). Since it's a first version, we expect it to have limitations, and plan to improve in next releases.
    Last edited by Cyan; 10th October 2017 at 20:55.

  42. Thanks (14):

    avitar (10th October 2017),Bulat Ziganshin (11th October 2017),hunman (10th October 2017),inikep (10th October 2017),Jarek (10th October 2017),load (10th October 2017),Mike (10th October 2017),przemoc (10th October 2017),schnaader (10th October 2017),Shelwien (10th October 2017),spwolf (10th October 2017),Stephan Busch (10th October 2017),terrelln (10th October 2017),tobijdc (15th October 2017)

  43. #299
    Member
    Join Date
    Sep 2011
    Location
    uk
    Posts
    238
    Thanks
    188
    Thanked 17 Times in 12 Posts
    1) Just downloaded the win32 version of this, and extracted the libs and dlls from zip file. Cannot remember whether it worked before. However when I type 'zstd' get entry point not found ... initialiseconditionvariable not located in in kernel32.dll So what else do I need to extract? Or is it because I'm running win xp and it isn't compiled from xp?

    2) using win64 version on win7 no dlls seem to be needed - or are they?
    4 processor amd with 8G memory, about 4G free. I'm compressing an 8.2G vbox vdi file.

    arc -mx 1.6G best overall takes 2000s
    rz 1.2G so that's the smallest but takes 4h ie 15000s !! too long

    zstd level 19 -T2 2.0G takes 1500s
    zstd 19 -T4 takes 1000s but crashes at end & does not save archive or set errorlevel

    with zstd level 15..19 I've tried --long but always crashes. Any idea why? --ultra level 20 crashes too, out of memory.
    Any way I can make it smaller?

    3) when there is memory crash, looks as if zstd does not set errorlevel

    TIA for any help. j
    Last edited by avitar; 10th October 2017 at 21:30.

  44. #300
    Member FatBit's Avatar
    Join Date
    Jan 2012
    Location
    Prague, CZ
    Posts
    190
    Thanks
    0
    Thanked 36 Times in 27 Posts

    Various (s)reps for your test

    Various (s)reps for your tests. I hope that Mr. Bulat will agree.

    Best regards,

    FatBit
    Attached Files Attached Files

  45. Thanks:

    Bulat Ziganshin (11th October 2017)

Page 10 of 14 FirstFirst ... 89101112 ... LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •