Why does BWT beat LZMA/LZMA2?

[msat59]

Hi everybody.

I don't know why BWT beats LZMA sometimes. I know that BWT sometimes works better on text files (eg enwiki8 ), but I don't get it why it outperforms LZMA while compressing Photoshop PSD files.

I attached a sample PSD file. For this one (and most of the files with a BW layer and a RGB layer), Rings Archiver (of Nania Francesco) and Nanozip Optimum 1&2 compress better. Here are the commands used in FreeArc.

LZMA:    9rep+mm:1*8+9rep+9xb:273:lc8:pb0:lp1
NZ 0.9:  9rep+9rep+nz9:co:m2g:t1:p1
Rings:   9rep+9rep+ringsx86:m6

Compressor                     Size 
-------------                   --------
LZMA                            5,181,956 
NZ 0.9 Optimum 1                4,684,064  
Ringsx86                        4,501,542

Is there anyway to improve the compression in LZMA?
I don't prefer to use an external compressor in FreeArc as LZMA mostly compresses better than BWT.

Thanks

[Shelwien]

Well, with

mm.exe -c1 -w8 Ladybug.psd 3
lzma.exe e 3 Ladybug.lzma1 -a1 -d25 -fb273 -mc999 -lc8 -lp0 -pb0 -mfbt4 -mt2

I'm getting 5,090,190 like this, so better results _are_ possible.
Also maybe its not an lzma problem, but simply these compressors have better filters than Bulat's mm?

[msat59]

Well, with

mm.exe -c1 -w8 Ladybug.psd 3
lzma.exe e 3 Ladybug.lzma1 -a1 -d25 -fb273 -mc999 -lc8 -lp0 -pb0 -mfbt4 -mt2

I'm getting 5,090,190 like this, so better results _are_ possible.
Also maybe its not an lzma problem, but simply these compressors have better filters than Bulat's mm?

Thanks.

So you mean nothing cannot be done to improve the compression of FreeArc internal LZMA?

What is that mm.exe? May you attach it here?

[Shelwien]

> So you mean nothing cannot be done to improve the compression of FreeArc internal LZMA?

You can use lp0 instead of lp1 at least.

> What is that mm.exe? May you attach it here?

http://freearc.org/download/research/mm11.zip

I just used standalone programs instead of freearc, it should be the same.

[jethro]

Thanks.

So you mean nothing cannot be done to improve the compression of FreeArc internal LZMA?

What is that mm.exe? May you attach it here?

for Bitmaps and LZMA you should use delta filter and parameters like:
0=delta:3 1=lzma:lc=7:lp=0: pb=0:fc=273:x=9 (remove the space after lp=0)

EDIT: I guess it is one channel (don't have Photoshop) then change delta:3 to delta:1. For this file lc=8 works best:

5*089*712 bytes

[msat59]

Thank you all.

The new size is 5,091,005 using:

/$psd=mm:1*8+9rep+9xb:273:lc8:pb1:lp0

but still far from BWT.

[Jyrki Alakuijala]

I don't know why BWT beats LZMA sometimes.

BWT works as a very long byte-based context. LZMA has a shorter but more complex context (+ LZ77ish modeling).

When the data benefits from a long byte-based context, BWT can be a better fit than advanced LZ77.

[msat59]

So Bulat needs to include the BWT code in FreeArc Next !


Isn't there any BWT preprocessor, not compressor?

[Shelwien]

You're not comparing lzma vs bwt here, but whole chains of filters + codecs.
And its not even certain that its impossible to tweak some MM options to get better results.
Also, neither BWT nor lzma are good codecs for images, at least use FLIF or something.

[xezz]

So Bulat needs to include the BWT code in FreeArc Next!

Already included BWT code(grzip)

[msat59]

Already included BWT code(grzip)

I meant as a preprocessor; however why doesn't the size of compressed file by grzip (5,122,656) go below LZMA, like the other two BWT compressors?

I think Nania should describe it.

EDIT:
I got 4,864,285 with mm:1*8+9rep+grzip. Now it's smaller than LZMA, but still bigger than other BWTs.

Also, neither BWT nor lzma are good codecs for images, at least use FLIF or something.

I don't think that it is possible to compress a PSD file with FLIF; however I wanted to find a solution using only FreeArc, not an external compressor. Nania's MCM gives better ratio and the (de)compression time is reasonable.

[xezz]

I got 4,864,285 with mm:1*8+9rep+grzip. Now it's smaller than LZMA, but still bigger than other BWTs.

grzip's maxmum bocksize is 8MB, so if file size is much larger than 8MB then maybe it can not beat NZ and Rings.
I hope grzip to implement large blocksize.

[msat59]

grzip's maxmum bocksize is 8MB, so if file size is much larger than 8MB then maybe it can not beat NZ and Rings.
I hope grzip to implement large blocksize.

Now I got it. Thanks

[Shelwien]

> I don't think that it is possible to compress a PSD file with FLIF

Actually it is kinda possible - convert it with imagemagick to .pnm or a set of bmps, then compress these with flif,
then decode flif, convert to psd with imagemagick and make a patch with xdelta from recoded image to original one.

I just tested it and flif compresses this file to 4,469,149 even as 4 separate images (psd layers) - the result
would be probably much better if we'd turn these layers into frames in the same file, or something.

But the problem is that .psd apparently has some RLE compression, and imagemagick uses it differently,
so only the psd produced by imagemagick has a chance of being losslessly compressed with this method atm.