Results 1 to 3 of 3

Thread: LZMA implemented on Hardware.

  1. #1
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    536
    Thanks
    238
    Thanked 90 Times in 70 Posts

    Lightbulb LZMA implemented on Hardware.

    Abstract

    Data transmission, storage and processing are the integral parts of today's information systems. Transmission and storage of huge volume of data is a critical task in spite of the advancements in the integrated circuit technology and communication. In order to store and transmit such a data as it is, requires larger memory and increased bandwidth utilization. This in turn increases the hardware and transmission cost. Hence, before storage or transmission the size of data has to be reduced without affecting the information content of the data. Among the various encoding algorithms, the Lempel Ziv Marcov chain Algorithm (LZMA) algorithm which is used in 7zip was proved to be effective in unknown byte stream compression for reliable lossless data compression. However the encoding speed of software based coder is slow compared to the arrival time of real time data. Hence hardware implementation is needed since number of instructions processed per unit time depends directly on system clock. The aim of this work is to implement the LZMA algorithm on SPARTAN 3E FPGA to design hardware encoder/decoder with reduces circuit size and cost of storage.

    Paper can be found here.


    What do you think? This is definitively something else...
    Although is far from finished.



    --------------------------------------------------------------------------------------------
    Edit: Another one, newer and faster implementation.

    In the era of big data, compression techniques are needed to improve data processing bandwidth and data storage efficiency. Compared with software-based compression techniques, hardware-based compression techniques can improve speed and reduce power consumption. LZMA (Lempel-Ziv-Markov chain algorithm) is a lossless compression technology, and its hardware implementation has broad application prospects. A novel high-performance implementation of the LZMA compression algorithm capable of processing up to 125 Mbit/s on a Virtex-6 field programmable gate array (FPGA) chip is proposed. A typical application and its compression performance for a specific data sample are then presented.

  2. Thanks:

    avitar (18th November 2014)

  3. #2
    Member
    Join Date
    Jan 2014
    Location
    Bothell, Washington, USA
    Posts
    695
    Thanks
    153
    Thanked 183 Times in 108 Posts
    It's interesting, but I have doubts about the performance. Spartan3E is old and I have a hard time picturing one running at 50MHz outperforming or even matching an average modern PC. Virtex-6 is better, since it is much bigger, faster and has features like PCIe. But you could definitely do better with something like Virtex Ultrascale because of the faster DDR interface.

    While FPGAs are wonderful at concurrent processing, they are not so good at performing sequential tasks quickly since the maximum clock rate is substantially less than 1 GHz. Compression results beyond tiny files would give a bit more insight.

  4. #3
    Member
    Join Date
    Sep 2008
    Location
    France
    Posts
    889
    Thanks
    482
    Thanked 279 Times in 119 Posts
    Interesting,
    I'm witnessing an increased interest into hardware compression very recently,
    and especially in China.
    I wouldn't be surprised it's all linked.

    But the results should be analyze with caution.
    I suspect the high speed is achieved by reducing the compression efficiency of LZMA,
    typically using some very small dictionary, and removing some hardware-unfriendly options present in the format.

    In such case, it does not directly compare with LZMA -9, which is what people have in mind when mentioning LZMA.

Similar Threads

  1. 2D fractal Haar wavelet transform (implemented)
    By Jarek in forum Data Compression
    Replies: 2
    Last Post: 12th September 2014, 15:50
  2. BWT + LZMA
    By Zelex in forum Data Compression
    Replies: 10
    Last Post: 21st July 2013, 11:44
  3. Hardware compression without software
    By BetaTester in forum Data Compression
    Replies: 0
    Last Post: 23rd January 2013, 20:05
  4. password cracking hardware
    By willvarfar in forum The Off-Topic Lounge
    Replies: 0
    Last Post: 27th June 2011, 10:52
  5. LZMA source
    By Shelwien in forum Data Compression
    Replies: 2
    Last Post: 29th March 2010, 18:45

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •