Results 1 to 5 of 5

Thread: tensorflow-compress

  1. #1
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    248
    Thanks
    113
    Thanked 122 Times in 71 Posts

    tensorflow-compress

    I have released a new open source compression program: https://github.com/byronknoll/tensorflow-compress

    It performs compression using LSTM and can run on GPUs. It is made using Colab, so can be run through a web browser. The v1 results are not very impressive, but I am planning to continue working on performance and compression rate improvements.

  2. Thanks (7):

    Bulat Ziganshin (23rd July 2020),comp1 (23rd July 2020),Cyan (29th July 2020),Darek (22nd July 2020),Hakan Abbas (22nd July 2020),Mauro Vezzosi (27th July 2020),schnaader (22nd July 2020)

  3. #2
    Member
    Join Date
    Jun 2020
    Location
    Belarus
    Posts
    6
    Thanks
    6
    Thanked 0 Times in 0 Posts
    Have noticed that test results were placed to LTCB
    http://mattmahoney.net/dc/text.html

  4. #3
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    277
    Thanks
    116
    Thanked 159 Times in 116 Posts
    Quote Originally Posted by byronknoll View Post
    It performs compression using LSTM and can run on GPUs.
    Quote Originally Posted by Shelwien View Post
    Yet another problem is about GPU data errors. Like, did you ever see random glitches on screen in games?
    Electronics are not 100% safe actually, there's simply a low enough error margin, like 10^-20 or so.
    Which is why there's ECC memory for servers, for example.
    But with enough load and processed data errors are bound to appear.
    And while in games it would be just harmless rendering glitches, in compression it would usually break
    all the data after the point of error.
    Is it safe to use GPU?
    I never tested NN GPUs libraries because I supposed was unsafe for online learning.

  5. #4
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    248
    Thanks
    113
    Thanked 122 Times in 71 Posts
    Quote Originally Posted by Mauro Vezzosi View Post
    Is it safe to use GPU?
    I never tested NN GPUs libraries because I supposed was unsafe for online learning.
    I think so? From my testing so far I have never seen any files fail validation after decompression. I used this page to get some advice on GPU determinism: https://github.com/NVIDIA/framework-determinism

  6. #5
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    846
    Thanks
    242
    Thanked 309 Times in 184 Posts
    Quote Originally Posted by Mauro Vezzosi View Post
    Is it safe to use GPU?
    I never tested NN GPUs libraries because I supposed was unsafe for online learning.
    Nothing is guaranteed with enough scale.

    This is why there can be block-based compression with checksums and retries for decompression. When compression is unsafe, then decompressing and checking for checksum immediately after compression is an option.

Similar Threads

  1. smaller compress
    By Obama in forum Random Compression
    Replies: 2
    Last Post: 5th September 2019, 11:18
  2. Compress company
    By Obama in forum Random Compression
    Replies: 2
    Last Post: 12th August 2019, 17:17
  3. MS COMPRESS v1.4
    By comp1 in forum Download Area
    Replies: 3
    Last Post: 19th May 2019, 07:27
  4. how to compress these numbers better?
    By Shelwien in forum Data Compression
    Replies: 8
    Last Post: 16th March 2019, 03:52
  5. Compress-LZF
    By spark in forum Data Compression
    Replies: 2
    Last Post: 16th October 2009, 00:08

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •