Results 1 to 2 of 2

Thread: BriefLZ

  1. #1
    Member jibz's Avatar
    Join Date
    Jan 2015
    Location
    Denmark
    Posts
    124
    Thanks
    106
    Thanked 71 Times in 51 Posts

    BriefLZ

    Since the BriefLZ 1.2.0 thread disappeared, here is a new one!

    I've just pushed BriefLZ 1.3.0 which includes the forwards binary tree parser (btparse) which was in the latest bcrush and blz4. It improves the speed of --optimal on many types of data, at the cost of using more memory. The format is still backwards compatible with BriefLZ 1.0.0.

    Code:
        enwik8        blzpack-1.2.0 --optimal -b100m     30,496,733   30 hours
        silesia.tar   blzpack-1.2.0 --optimal -b205m     63,838,305   about a week
    
        enwik8        blzpack-1.3.0 --optimal -b100m     30,496,733   95 sec
        silesia.tar   blzpack-1.3.0 --optimal -b205m     63,836,210   4 min
        enwik9        blzpack-1.3.0 --optimal -b1g      252,991,647   10.5 hours
    Not sure why the result for silesia.tar from 1.2.0 from two years ago is slightly higher, but not going to rerun it. If anyone has a machine with 32GiB of RAM, I would love to hear how long --optimal -b1g on enwik9 takes, because the results for this machine (8GiB RAM) includes swapping.

    Attached is a Windows 64-bit executable, and the source is at https://github.com/jibsen/brieflz
    Attached Files Attached Files

  2. Thanks:

    Mike (15th February 2020)

  3. #2
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,026
    Thanks
    103
    Thanked 410 Times in 285 Posts
    Quote Originally Posted by jibz View Post
    If anyone has a machine with 32GiB of RAM, I would love to hear how long --optimal -b1g on enwik9 takes
    252,991,647 bytes, 4628.815 sec. (1 hour 17 min) - 3.683 sec., blzpack -9 --optimal -b1g (v1.3.0)

  4. Thanks:

    jibz (16th February 2020)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •