Results 1 to 2 of 2

Thread: Snappy Compression for large number of small files

  1. #1
    Join Date
    Mar 2013
    Thanked 0 Times in 0 Posts

    Red face Snappy Compression for large number of small files

    Hi all,

    I am doing a backup ans restore project on C++ .

    I need to back one million small files ranging from 5kb . I follow the steps below,

    1. Read files upto 1 GB and save in a buffer and read 7200 bytes from buffer.
    2. Compress it.
    3. And send to server.

    I m trying Snappy algorithm for compression. Can i get fixed length of bytes(say 1.5MB) as a compressed output each time . Using Zlib's dictionary i was able to attain that. So that while restoring the backed up files , i will be able to give 1.5 MB and decompress it. But Zlib seems to be slower , so i opted for snappy algorithm. How can i implement the dictionary concept in snappy algorithm effectively. ??


    Last edited by Selvaraj; 29th March 2013 at 10:40.

  2. #2
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Melbourne, Florida, USA
    Thanked 798 Times in 489 Posts
    I think zlib saves the compressor state information when it reaches end of output and allows you to resume from that state. There might be some performance penalty in Snappy. Try compressing 64K at a time until your output buffer is nearly full. Snappy uses a small window so it should not lose too much compression this way.

Similar Threads

  1. Compression via Number Factorisation
    By MiY4Gi in forum Random Compression
    Replies: 14
    Last Post: 20th July 2012, 19:51
  2. Replies: 23
    Last Post: 17th September 2011, 13:12
  3. Replies: 33
    Last Post: 27th August 2011, 06:13
  4. Small dictionary prepreprocessing for text files
    By Matt Mahoney in forum Data Compression
    Replies: 40
    Last Post: 23rd June 2011, 07:04
  5. Google released Snappy compression/decompression library
    By Sportman in forum Data Compression
    Replies: 11
    Last Post: 16th May 2011, 13:31

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts