fast opensource multithread stidn packer

[fcorbelli]

i am searching for a freebsd compressor that con take data from pipe, with high parallelism (say 8 real threads) and high performances with low clock.
currently i use pbzip2, but maybe there is something faster.
data is zfs binary dump, so not much compressable.
ideally i need 150/200MB/s with 6-8 threads at 2GHz

[thometal]

7z with chunks in fastest mode. detects also when data is incompressable and just saves

maybe zpipe ? but idont know if its multithreading

[m^2]

FreeArc doesn't build on FreeBSD.
Plzip?

The question is if you need a strong compressor at all. The best place to compress may (or may not) be zfs itself. Aside from that, if most of your data is already compressed, something like zstd may give you all the strength you need while using just 1 core for negligible strength difference.

[fcorbelli]

Thanks to all answers.
I do not need a strong compressor, because it's simply infeasible to have enough CPU power to process a stream at sustained 200MB/s.

The data is a stream of a zfs filesystem (or better from a zfs snapshot), so it's almost all already compressed; in my case I am trying to gain say 10 or 20% when writing to the target system, because mechanical drive cannot write much more than 150-180 MB/s sustained.

For those not familiar with zfs is possible to backup a "live" system (something like acronis etc) sending an entire disk (snapshot of a zpool, to be precise) to a file (or a pipe).

Using very fast source (Xeon-SSD), high speed link (10Gb) the bottleneck became the target write bandwidth (aka the drive on the NAS), so every 1% of reduction will cut the time needed to complete a TB+ job.

Xeon-servers tipically does not have much CPU power (low clock ~2GHz), but many physical cores (8 in my case), and plenty of RAM (128GB), and very fast disk (SSD or even better M2).

I will try pigz and plzip

[m^2]

You may also try Bulat's rep/srep, either alone or in tandem with something else. Not sure if they support pipe IO though.

[fcorbelli]

You may also try Bulat's rep/srep, either alone or in tandem with something else. Not sure if they support pipe IO though.

for now the winner is pigz. real fast indeed with many cored