Page 12 of 12 FirstFirst ... 2101112
Results 331 to 340 of 340

Thread: Zstandard

  1. #331
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    783
    Thanks
    64
    Thanked 273 Times in 191 Posts
    Zstd v1.4.0 64-bits:
    https://github.com/facebook/zstd/releases

    Input:
    enwik9

    Output:
    357,515,962 bytes, 2.361 sec. - 1.130 sec., -1
    329,419,520 bytes, 3.353 sec. - 1.235 sec., -2
    314,012,348 bytes, 4.333 sec. - 1.287 sec., -3
    308,269,618 bytes, 4.584 sec. - 1.346 sec., -4
    302,075,641 bytes, 7.075 sec. - 1.353 sec., -5
    295,590,745 bytes, 9.556 sec. - 1.338 sec., -6
    285,397,982 bytes, 13.247 sec. - 1.285 sec., -7
    281,180,106 bytes, 16.799 sec. - 1.241 sec., -8
    278,757,548 bytes, 24.206 sec. - 1.242 sec., -9
    273,782,728 bytes, 29.656 sec. - 1.246 sec., -10
    271,392,063 bytes, 38.660 sec. - 1.256 sec., -11
    269,321,220 bytes, 59.222 sec. - 1.236 sec., -12
    266,022,487 bytes, 75.572 sec. - 1.212 sec., -13
    261,574,115 bytes, 101.186 sec. - 1.225 sec., -14
    258,869,397 bytes, 138.761 sec. - 1.232 sec., -15
    250,212,437 bytes, 156.968 sec. - 1.222 sec., -16
    242,902,736 bytes, 232.295 sec. - 1.205 sec., -17
    239,765,452 bytes, 273.279 sec. - 1.245 sec., -18
    235,698,881 bytes, 368.606 sec. - 1.277 sec., -19
    226,024,466 bytes, 467.710 sec. - 1.428 sec., -20
    220,222,797 bytes, 529.841 sec. - 1.483 sec., -21
    215,032,608 bytes, 569.795 sec. - 1.507 sec., -22

  2. The Following 4 Users Say Thank You to Sportman For This Useful Post:

    avitar (28th April 2019),Bulat Ziganshin (28th April 2019),Cyan (29th April 2019),Jango (30th April 2019)

  3. #332
    Member
    Join Date
    May 2007
    Location
    Poland
    Posts
    89
    Thanks
    8
    Thanked 4 Times in 4 Posts
    How to use dictionary with ZSTD?
    I tried:
    Code:
    zstd  BJ_all_Corr.csv -D dict.1 -o dict.zstd
    zstd: cannot use BJ_all_Corr.csv as an input file and dictionary
    How do i tell zstd which file is the trained dictonary (dict.1 here)?

  4. #333
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,269
    Thanks
    200
    Thanked 985 Times in 511 Posts
    zstd dictionary file is not raw, you have to build it first:
    Code:
    Dictionary builder :
    --train ## : create a dictionary from a training set of files
    --train-cover[=k=#,d=#,steps=#,split=#] : use the cover algorithm with optional args
    --train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#] : use the fast cover algorithm with optional args
    --train-legacy[=s=#] : use the legacy algorithm with selectivity (default: 9)
     -o file : `file` is dictionary name (default: dictionary)
    --maxdict=# : limit dictionary to specified size (default: 112640)
    --dictID=# : force dictionary ID to specified value (default: random)

  5. #334
    Member
    Join Date
    May 2007
    Location
    Poland
    Posts
    89
    Thanks
    8
    Thanked 4 Times in 4 Posts
    Yes, this is the dict.1 file
    Code:
    zstd --train .\train\*  -o dict.1
    Trying 5 different sets of parameters
    k=1998
    d=8
    f=20
    steps=4
    split=75
    accel=1
    Save dictionary of size 112640 into file dict.1

  6. #335
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    863
    Thanks
    458
    Thanked 257 Times in 105 Posts
    Hi @Jethro

    This command line you present should have worked.

    zstd FILE -D dict -o dest
    is a usual construction that is known and well tested.

    I can't explain from this snippet why it would not work for you ....

  7. #336
    Member
    Join Date
    May 2007
    Location
    Poland
    Posts
    89
    Thanks
    8
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by Cyan View Post
    Hi @Jethro

    This command line you present should have worked.



    is a usual construction that is known and well tested.

    I can't explain from this snippet why it would not work for you ....
    Thanks Cyan
    *** zstd command line interface 64-bits v1.4.2, by Yann Collet ***
    Win 10

  8. #337
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    52
    Thanks
    27
    Thanked 11 Times in 8 Posts
    I have got a question:
    Is Zstandard capable of compressing very large files, e.g. files of 40 Gbytes each, or is there a file size limit?

  9. #338
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,269
    Thanks
    200
    Thanked 985 Times in 511 Posts
    No filesize limit normally. Zstd API (zstd.h) doesn't even work with files, but rather streams of unknown length.

    A custom file format which uses zstd for compression can easily have such limits though.

  10. The Following User Says Thank You to Shelwien For This Useful Post:

    WinnieW (12th October 2019)

  11. #339
    Member
    Join Date
    Jan 2017
    Location
    Germany
    Posts
    52
    Thanks
    27
    Thanked 11 Times in 8 Posts
    I can confirm there is no problem. I compressed a file of 38 Gbyte of size using the official 64 Bit Windows command line binary. Verified the file integrity using SHA1 checksums. Original file and decompressed file were bit identical.

  12. The Following User Says Thank You to WinnieW For This Useful Post:

    Cyan (13th October 2019)

  13. #340
    Member
    Join Date
    May 2007
    Location
    Poland
    Posts
    89
    Thanks
    8
    Thanked 4 Times in 4 Posts
    Hi @Jethro

    This command line you present should have worked.

    zstd FILE -D dict -o dest
    is a usual construction that is known and well tested.

    I can't explain from this snippet why it would not work for you ....
    https://github.com/facebook/zstd/issues/1817

  14. The Following User Says Thank You to jethro For This Useful Post:

    Shelwien (13th October 2019)

Page 12 of 12 FirstFirst ... 2101112

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •