Results 1 to 8 of 8

Thread: tensorflow-compress

  1. #1
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    249
    Thanks
    113
    Thanked 123 Times in 72 Posts

    tensorflow-compress

    I have released a new open source compression program: https://github.com/byronknoll/tensorflow-compress

    It performs compression using LSTM and can run on GPUs. It is made using Colab, so can be run through a web browser. The v1 results are not very impressive, but I am planning to continue working on performance and compression rate improvements.

  2. Thanks (7):

    Bulat Ziganshin (23rd July 2020),comp1 (23rd July 2020),Cyan (29th July 2020),Darek (22nd July 2020),Hakan Abbas (22nd July 2020),Mauro Vezzosi (27th July 2020),schnaader (22nd July 2020)

  3. #2
    Member
    Join Date
    Jun 2020
    Location
    Belarus
    Posts
    7
    Thanks
    6
    Thanked 1 Time in 1 Post
    Have noticed that test results were placed to LTCB
    http://mattmahoney.net/dc/text.html

  4. #3
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    278
    Thanks
    116
    Thanked 160 Times in 117 Posts
    Quote Originally Posted by byronknoll View Post
    It performs compression using LSTM and can run on GPUs.
    Quote Originally Posted by Shelwien View Post
    Yet another problem is about GPU data errors. Like, did you ever see random glitches on screen in games?
    Electronics are not 100% safe actually, there's simply a low enough error margin, like 10^-20 or so.
    Which is why there's ECC memory for servers, for example.
    But with enough load and processed data errors are bound to appear.
    And while in games it would be just harmless rendering glitches, in compression it would usually break
    all the data after the point of error.
    Is it safe to use GPU?
    I never tested NN GPUs libraries because I supposed was unsafe for online learning.

  5. #4
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    249
    Thanks
    113
    Thanked 123 Times in 72 Posts
    Quote Originally Posted by Mauro Vezzosi View Post
    Is it safe to use GPU?
    I never tested NN GPUs libraries because I supposed was unsafe for online learning.
    I think so? From my testing so far I have never seen any files fail validation after decompression. I used this page to get some advice on GPU determinism: https://github.com/NVIDIA/framework-determinism

  6. #5
    Member
    Join Date
    Jun 2015
    Location
    Switzerland
    Posts
    900
    Thanks
    246
    Thanked 326 Times in 199 Posts
    Quote Originally Posted by Mauro Vezzosi View Post
    Is it safe to use GPU?
    I never tested NN GPUs libraries because I supposed was unsafe for online learning.
    Nothing is guaranteed with enough scale.

    This is why there can be block-based compression with checksums and retries for decompression. When compression is unsafe, then decompressing and checking for checksum immediately after compression is an option.

  7. #6
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    371
    Thanks
    55
    Thanked 63 Times in 51 Posts
    http://mattmahoney.net/dc/text.html#1639
    tensorflow compress v2 is released and at 5th position
    it is great!!!

  8. #7
    Member JamesWasil's Avatar
    Join Date
    Dec 2017
    Location
    Arizona
    Posts
    107
    Thanks
    96
    Thanked 19 Times in 18 Posts
    Quote Originally Posted by Jyrki Alakuijala View Post
    Nothing is guaranteed with enough scale.

    This is why there can be block-based compression with checksums and retries for decompression. When compression is unsafe, then decompressing and checking for checksum immediately after compression is an option.
    True. Using a GPU for everything would be unsafe without error correction or checksums. But maybe the CPU can generate an interrupt for checksum after block equivalents are emitted from the GPU instead to fix that?

    It wouldn't be entirely GPU based anymore, but would be safe enough to where the margin for error becomes negligible and the speed advantage is still at least 50% to 75% there with the routine checks for safety and data integrity.

    I was going to suggest a second GPU to do these checksums but that would be overkill. Any CPU should be able to handle the task reasonably to review a CRC block and make sure it is correct before saving it to disk.

    It may even be possible to multithread several GPUs this way if it is successful and safe, and then the CPU can handle and process additional checksums easily (since that would be it's task and not other major instructions) to where multiple outputs from a second or third GPU working in parallel to it could be safely evaluated and increase compression times drastically for larger data. That would make up for any speed hit between the CPU and GPU for safety and sanity checks, and then some, since the additional GPU would cover for the latency of the first if more than one is used at the same time on a data stream.

    Might be best to test it out with 1 GPU to see how well it works, then 2 or more later if the approach this way is successful. What do you think?

  9. #8
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    200
    Thanks
    136
    Thanked 13 Times in 12 Posts
    can someone share binary of tensorflow or deepzip?

Similar Threads

  1. smaller compress
    By Obama in forum Random Compression
    Replies: 2
    Last Post: 5th September 2019, 11:18
  2. Compress company
    By Obama in forum Random Compression
    Replies: 2
    Last Post: 12th August 2019, 17:17
  3. MS COMPRESS v1.4
    By comp1 in forum Download Area
    Replies: 3
    Last Post: 19th May 2019, 07:27
  4. how to compress these numbers better?
    By Shelwien in forum Data Compression
    Replies: 8
    Last Post: 16th March 2019, 03:52
  5. Compress-LZF
    By spark in forum Data Compression
    Replies: 2
    Last Post: 16th October 2009, 00:08

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •