7-zip parameter optimization

[dado023]

just did quick compression test...used intel Lan drivers:


seems like normal 7-zip does best compression atm.....if you dont consider time as a deciding factor.

Is there a way to go beyond level 9 with 7-Zip ZS(LZMA2,fast)? to increase compression even more?

[Shelwien]

There're some ways:
1. mx9 uses 64M window, you can specify up to 1.5G manually via -md
2. Use lzma rather than lzma2, and no more than 1-2 threads
3. Add -myx=9 switch
4. manually specify more efficient BCJ2 scheme for exes; https://encode.su/threads/15-7-Zip?p...ll=1#post57538
5. optimize lc/pb/lp and fb settings
6. Recompile lzma with tweaked parsing optimization options, like "#define kNumOpts (1 << 12)" to 1<<15

[dado023]

when i try to use -myx=9 i get error saying:

I am using it like this: ...how do i add -myx=9 switch properly?

Edit: i am using 7-Zip 19.00 64bit

[Shelwien]

Just read the help on that parameters string... It says "omit -m", so write yx=9 there.

[dado023]

interestingly, yx=9 produces larger file size, by around 150kB
If anyone wants to give it a try, here is file for testing: LAN_Intel_19030_LZMA_ultra_1536MB_wordSize273_soli d_1t.7z

[Shelwien]

7z a -mx=9 -bb3 -mqs=on ^
-m0=bcj2:d24 ^
-m1=lzma:mt2:d=384M:lc8:pb2:lp0:fb273:mc999 ^
-m2=delta:4 ^
-m3=lzma:mt2:d24:lc0:pb2:lp2:fb273 ^
-m4=lzma:mt2:d24:lc0:pb2:lp2:fb273 ^
-m5=lzma2:mt2:d24:lc0:pb0:lp0:fb273 ^
-mb00s0:1 -mb00s1:2 -mb00s2:4 -mb00s3:5 -mb02s0:3 ^
DP_LAN_Intel_19030.7z DP_LAN_Intel_19030

21,852,665 DP_LAN_Intel_19030_LZMA_ultra_1536MB_wordSize273_solid_1t.7z0
21,291,826 DP_LAN_Intel_19030.7z

[Conor]

In the 7-Zip file manager you can also add a=3 which should improve it a bit. Fast LZMA2 rarely compresses as well as LZMA2 at equal window sizes because the match-finder is a block algorithm.

-ma=3 applies on the command line but it also sets the window size by the compression level: x1 = 1 Mb, x9 = 256 Mb

[dado023]

@Shelwien i have never tried all those switches at once....do i dropout "m" on all like for myx....or this is only exclusive for command line only?

@Conor, just to clear for my self....is window size actually dictionary size? or those two are completely different parameters?

edit: how do i make forum user names active in a post?

[Shelwien]

I tried inserting "x=9 qs=on 0=bcj2:d24 1=lzma:mt2:d=384M:lc8:pb2:lp0:fb273:mc999 2=delta:4 3=lzma:mt2:d24:lc0:pb2:lp2:fb273 4=lzma:mt2:d24:lc0:pb2:lp2:fb273 5=lzma2:mt2:d24:lc0:pb0:lp0:fb273 b00s0:1 b00s1:2 b00s2:4 b00s3:5 b02s0:3"
into 7zFM parameters and it worked. "-bb3" only affects console output, so you can skip it.

> is window size actually dictionary size? or those two are completely different parameters?

7-zip/lzma calls the data that can be copied by LZ decoder as "dictionary".
Its not exactly wrong, but with time, new things were added (like dictionary preprocessing and external dictionary for LZ),
so now its better to use "window" and "window size" for this.

> how do i make forum user names active in a post?

With bbcode tags, but just don't - this forum engine is old and doesn't have these features really.

[dado023]

I tried inserting "x=9 qs=on 0=bcj2:d24 1=lzma:mt2:d=384M:lc8b2:lp0:fb273:mc999 2=delta:4 3=lzma:mt2:d24:lc0b2:lp2:fb273 4=lzma:mt2:d24:lc0b2:lp2:fb273 5=lzma2:mt2:d24:lc0b0:lp0:fb273 b00s0:1 b00s1:2 b00s2:4 b00s3:5 b02s0:3"
into 7zFM parameters and it worked. "-bb3" only affects console output, so you can skip it.

> is window size actually dictionary size? or those two are completely different parameters?

7-zip/lzma calls the data that can be copied by LZ decoder as "dictionary".
Its not exactly wrong, but with time, new things were added (like dictionary preprocessing and external dictionary for LZ),
so now its better to use "window" and "window size" for this.

> how do i make forum user names active in a post?

With bbcode tags, but just don't - this forum engine is old and doesn't have these features really.

As you recommend i wont use bbcode tags.

Regarding parameters you provided for 7zFM, this is by far the highest compression with 7zip i ever saw:

I feel i discovered something completely new , exciting...i just wonder how come 7zip doesnt have this setting maybe under different name, for example "Extreme", because even the compression speed is quite acceptable


Another question: are these parameters good for all types of files(content) or you tuned this specifically for these hardware drivers?

[Shelwien]

> or you tuned this specifically for these hardware drivers?

Most of it would be useful for any exes. Except for BCJ2:d parameter, which depends on maximum exe size in archive - http://nishi.dreamhosters.com/u/method.htm#BCJ2
and similarly the :d parameters in all lzma streams.

Also, qs=on enables extension sort, which is usually better for compression, but NTFS uses name sort, so qs=on is kinda hard for the filesystem (fragmentation etc).

As to lzma, in theory lzma2 should be better, but unfortunately along with improvements, Igor also decided to put a limit on lc+lp value
for speed optimization, so best compression is now only possible for lzma, but not lzma2.

[dado023]

you mention LZMA can achieve better compression......do i use same parameters you provided before in 7zFM with just LZMA method selected in the dialog box.....or different parameters are needed in this case?

[dado023]

@Conor, i tried using a=3 but it doesn't accept this parameter in 7zFM

[Shelwien]

> you mention LZMA can achieve better compression......
> do i use same parameters you provided before in 7zFM with just LZMA method selected in the dialog box.....
> or different parameters are needed in this case?

Parameters can be tweaked based on data type.
As example, for tables of 32-bit values, its better to use lp2:pb2 and lower lc, maybe lc0 (like what I did for bcj2 addr streams).
For text and exes its usually best to use lc8, but there're cases where different lc value can be optimal
(default is lc3; lc0 means using order0 CM for literals, lc8 means order1).
Also, if lc+lp<=4, its better to use lzma2.

Btw, .7z format also supports ppmd codec, which has its own parameters and can compress texts better than lzma.

[dado023]

after reading this https://www.sami-lehtinen.net/blog/l...nd-compression i am even more interested in "window size"...if i understand correct, windows size works as a sort of deduplication technique

@Shelwien you mention windows size can be up to 1.5G using -md parameter, right? ...if m is omitted i assume its actually d

"x=9 qs=on 0=bcj2:d24 1=lzma:mt2:d=384M:lc8:pb2:lp0:fb273:mc999 2=delta:4 3=lzma:mt2:d24:lc0:pb2:lp2:fb273 4=lzma:mt2:d24:lc0:pb2:lp2:fb273 5=lzma2:mt2:d24:lc0:pb0:lp0:fb273 b00s0:1 b00s1:2 b00s2:4 b00s3:5 b02s0:3"

d24 ? or d=384M ? ...do i replace those number in above parameter set with for example d31 and d=1536M to make it work and get a bit better compression ratio?

[Shelwien]

> if i understand correct, window size works as a sort of deduplication technique

Technically, deduplication methods are specialized compression algorithms tuned
to use less memory comparing to LZ with similar window size.
The most popular one is probably srep by Bulat Ziganshin.
http://web.archive.org/web/201612231...arch/SREP.aspx
http://freearc.dreamhosters.com/srep393a.zip

> you mention windows size can be up to 1.5G using -md parameter, right? ...if m is omitted i assume its actually d

Yes, although its 7-zip that limits the "dictionary size" to 2^n and 3*2^n values.
lzma format supports match distances up to 4G, standard lzma encoder supports up to ~2000M due to double buffer.

> d24 ? or d=384M ?

d24 is 7-zip syntax for setting window size to 2^n bytes, ie 2^24=16M in this case.

> ...do i replace those number in above parameter set with for example d1024
> and d=1024M to make it work and get a bit better compression ratio?

d30 or d=1G or d=1024M would work, d1024 is probably not what you want.
Just read the 7-zip help: http://nishi.dreamhosters.com/u/meth...DictionarySize

But larger "dictionary" won't improve compression if its larger than data size.

[dado023]

@Shelwien

thank you so much for your reply. You are very informative.

>But larger "dictionary" won't improve compression if its larger than data size.
Now i understand,.....there is no point of using 1024M if all files together are for example 700M, right?
However, does it reduce compression if dict size is above total file size?


I am not sure how to use srep with 7zFM, if its usable at all with it....i guess that's out of the scope of 7z

[Shelwien]

> Now i understand,.....there is no point of using 1024M if all files together are for example 700M, right?

7-zip allows a limited set of dictionary sizes (because :d value has to be stored in archive),
so unfortunately its not as simple as setting dictionary size to data size.
So for 700M of data, you can use d=768M, but for 800M of data it might be better to use 1024M.

> However, does it reduce compression if dict size is above total file size?

No, but it might affect memory usage at compression.
lzma wants 10.5*d of memory _per instance_, so its easy enough to make it use more memory than you have.

> I am not sure how to use srep with 7zFM,
> if its usable at all with it....i guess that's out of the scope of 7z

7-zip supports external codecs, so its possible.
If it doesn't already exist, I can probably make one, but it would be better
to check if you need it first, by testing compression with standalone srep.

There's also freearc which has it integrated (along with lzma and other codecs).

[dado023]

@Shelwien

now i understand dict size and memory tradeoff much better. Thank you so much.

>If it doesn't already exist, I can probably make one, but it would be better

i don't need it personally as a must nor professionally, i am just learning a bit more about compression and how to improve it with better parameters, ...thanks to you, now i know how to compress at much higher ratios with 7zFM. I am grateful for that.
I am not sure how whole thing with external codecs works, but it would be awesome if you could improve compression ratio even more with srep combined with 7zFM, the whole world is always looking for higher compression

As for freearc, it kinda doesn't look maintained, so for long term, i don't want to have my files in that format

[zubzer0]

7-ZIP finetuner
http://krzyrio.blogspot.com/2017/01/...finetuner.html

automatic search for the best parameters, but alas, in 1 thread mode (search), because of what is slow.

[dado023]

@zubzer0

Does it work with latest 7zip 19.0 ?

[zubzer0]

@dado023

Yes, seems no problem. But as it was said, it lacks a multithreaded search ...
I tried to contact the author to contribute this idea, but he disappeared.
"7-ZIP finetuner" author https://encode.su/members/3822-Mr_KrzYch00 (Last Activity 18th September 2018 )

p.s. Perhaps someone else will take up writing alternatives to this program?