LZMA implemented on Hardware.

[Gonzalo]

Abstract

Data transmission, storage and processing are the integral parts of today's information systems. Transmission and storage of huge volume of data is a critical task in spite of the advancements in the integrated circuit technology and communication. In order to store and transmit such a data as it is, requires larger memory and increased bandwidth utilization. This in turn increases the hardware and transmission cost. Hence, before storage or transmission the size of data has to be reduced without affecting the information content of the data. Among the various encoding algorithms, the Lempel Ziv Marcov chain Algorithm (LZMA) algorithm which is used in 7zip was proved to be effective in unknown byte stream compression for reliable lossless data compression. However the encoding speed of software based coder is slow compared to the arrival time of real time data. Hence hardware implementation is needed since number of instructions processed per unit time depends directly on system clock. The aim of this work is to implement the LZMA algorithm on SPARTAN 3E FPGA to design hardware encoder/decoder with reduces circuit size and cost of storage.

Paper can be found here.


What do you think? This is definitively something else...
Although is far from finished.



--------------------------------------------------------------------------------------------
Edit: Another one, newer and faster implementation.

In the era of big data, compression techniques are needed to improve data processing bandwidth and data storage efficiency. Compared with software-based compression techniques, hardware-based compression techniques can improve speed and reduce power consumption. LZMA (Lempel-Ziv-Markov chain algorithm) is a lossless compression technology, and its hardware implementation has broad application prospects. A novel high-performance implementation of the LZMA compression algorithm capable of processing up to 125 Mbit/s on a Virtex-6 field programmable gate array (FPGA) chip is proposed. A typical application and its compression performance for a specific data sample are then presented.

[Kennon Conrad]

It's interesting, but I have doubts about the performance. Spartan3E is old and I have a hard time picturing one running at 50MHz outperforming or even matching an average modern PC. Virtex-6 is better, since it is much bigger, faster and has features like PCIe. But you could definitely do better with something like Virtex Ultrascale because of the faster DDR interface.

While FPGAs are wonderful at concurrent processing, they are not so good at performing sequential tasks quickly since the maximum clock rate is substantially less than 1 GHz. Compression results beyond tiny files would give a bit more insight.

[Cyan]

Interesting,
I'm witnessing an increased interest into hardware compression very recently,
and especially in China.
I wouldn't be surprised it's all linked.

But the results should be analyze with caution.
I suspect the high speed is achieved by reducing the compression efficiency of LZMA,
typically using some very small dictionary, and removing some hardware-unfriendly options present in the format.

In such case, it does not directly compare with LZMA -9, which is what people have in mind when mentioning LZMA.