Linux Kernel community phases out Bzip2 for source bundles

[moinakg]

A slightly old news but shows a shift towards LZMA within the open source community:
https://www.kernel.org/happy-new-year-and-good-bye-bzip2.html

The unfortunate thing is that XZ is very bad on performance. It is significantly slower than 7-Zip. I feel, performance aspect in XZ was forgotten in the name of file format and API standardization and code restructuring.

[Bulat Ziganshin]

why it's slower than 7-zip if both use the same lzma algo?

[moinakg]

The code is completely different from original 7-Zip implementation. I have found it to be quite slow in my tests: http://moinakg.github.io/pcompress/results1.html
I do not know the exact cause. Maybe it uses different parameters.

[seth]

I'm not really seeing a difference between xz and 7-zip in terms of decompression speed on my machines. Especially when compared to bzip2.

One of the big hurdles for 7zip in the *nix community is that the current commandline interface doesn't adhere to the POSIX standards. This makes it more difficult to insert into the full toolchains that are already in place.

~$ time unxz linux-3.13-rc7.tar.xz 

real    0m7.294s
user    0m6.716s
sys    0m0.576s

~$ time 7z x linux-3.13-rc7.tar.7z 

7-Zip [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,8 CPUs)

Processing archive: linux-3.13-rc7.tar.7z

Extracting  linux-3.13-rc7.tar

Everything is Ok

Size:       549806080
Compressed: 80265748

real    0m7.143s
user    0m6.564s
sys    0m0.576s

~$ time bunzip2 linux-3.13-rc7.tar.bz2 

real    0m15.859s
user    0m15.281s
sys    0m0.580s

[seth]

Testing compression speeds using the flags from your benchmark seems about the same on my machine. Maybe your version was compiled differently? Or maybe a strong machine dependency?

~$ time 7za a -t7z -mx6 linux-3.13-rc7.tar.7z linux-3.13-rc7.tar

7-Zip (A) [64] 9.20  Copyright (c) 1999-2010 Igor Pavlov  2010-11-18
p7zip Version 9.20 (locale=en_US.UTF-8,Utf16=on,HugeFiles=on,8 CPUs)
Scanning

Creating archive linux-3.13-rc7.tar.7z

Compressing  linux-3.13-rc7.tar      

Everything is Ok

real    3m9.577s
user    5m21.212s
sys    0m1.864s

~$ time xz -z -6 -k linux-3.13-rc7.tar -v
linux-3.13-rc7.tar (1/1)
  100 %        76.0 MiB / 524.3 MiB = 0.145   1.7 MiB/s       5:13             

real    5m13.160s
user    5m12.584s
sys    0m0.572s

[Bulat Ziganshin]

The code is completely different from original 7-Zip implementation. I have found it to be quite slow in my tests: http://moinakg.github.io/pcompress/results1.html
I do not know the exact cause. Maybe it uses different parameters.

it may be because it doesn't implement lzma multithreading. check CPU times. anyway, compression time isn't important for kernel.org purposes - they have classic compress-once-extract-many usage

[thometal]

it may be because it doesn't implement lzma multithreading. check CPU times. anyway, compression time isn't important for kernel.org purposes - they have classic compress-once-extract-many usage

yes i agree compressiontiom is nearly doubled in your tests, but i though that was already implemented........hmmhm

[Mangix]

Multi-threading is implemented in the latest xz alpha(5.1.3) but not in the stable.

[Matt Mahoney]

http://moinakg.github.io/pcompress/results1.html

I thought I would test the two concatenated copies of silesia.tar on zpaq, but I get 423,896,064 bytes for the original size, not 413,224,960. Anyway, on a 2 GHz T3200 (2 cores, Win32), which is slower than their test machine:

136,470,367 silesia2.tar.gz 37.6 s
109,140,158 silesia2.tar.bz2 144.3 s
99,458,006 silesia2.7z 262.6 s
69,603,834 silesia-1.zpaq 21.3 s
67,175,106 silesia-2.zpaq 38.7 s
59,226,485 silesia-3.zpaq 62.5 s
48,343,287 silesia-4.zpaq 163.9 s
43,598,975 silesia-5.zpaq 337.8 s
40,138,239 silesia-6.zpaq 1161.0 s

[caveman]

The code is completely different from original 7-Zip implementation.

Is it related to LZMA2?

[thometal]

no xz tools make just did programmed from scratch according spec