7-Zip binaries using the Fast LZMA2 lib

[Conor]

Attached are 64-bit 7-Zip binaries which use the new library. I'll put the source on github soon. They are v 17.01 because I haven't merged v 18 yet.

In the GUI, the initial word size setting may be loaded from the registry. It's best to initialize by changing the level because the default values are higher than normal 7z.

The compression levels use the same dictionary size as normal 7z, which results in a little less compression because the MF is a block algorithm. Bumping the level up one tends to even it out. Size at level 8 & 9 is higher than normal 7z at 128Mb.

Note the low multithread memory usage

EDIT: attached an updated version due to a bug in my 7-zip stream handler which caused it to infinite loop after the library finished compressing, if dict > input size. This version has smoother progress updates too.

EDIT 2: Fixed a crash when progress is set to NULL, e.g. with filters like BCJ2

EDIT 3: It has now passed plenty of automated testing and is released on github for x86 and x64: https://github.com/conor42/7-Zip-FL2/releases/

[Bulat Ziganshin]

imho, it will be better to put it into your original thread. otherwise, a few months later it will become a mess, Or, crosslink these two topics in their header posts

About modified 7-zip package - i propose to publish your codec as separate one, distinct from original lzma2. This will allow people to easily drop your dll into mainline 7-zip installations as well as other programs (freearc/powerarchiver can compress to .7z, RAR/WinZip can extract .7z and so on), compare lzma2 and your codec and choose which one to use for each run. It's especially important taking into account that your codec isn't absolute win so sometimes original lzma2 is preferable

I.e. instead of providing modified 7-zip package i would prefer if you will provide just libflzma2-x64.dll which one can drop into any 7z.dll-based program and immediately get access to your codec without sacrificing the original lzma2

If you don't know how to arrange that - i can help

[Gonzalo]

imho, it will be better to put it into your original thread. otherwise, a few months later it will become a mess, Or, crosslink these two topics in their header posts

About modified 7-zip package - i propose to publish your codec as separate one, distinct from original lzma2. This will allow people to easily drop your dll into mainline 7-zip installations as well as other programs (freearc/powerarchiver can compress to .7z, RAR/WinZip can extract .7z and so on), compare lzma2 and your codec and choose which one to use for each run. It's especially important taking into account that your codec isn't absolute win so sometimes original lzma2 is preferable

I.e. instead of providing modified 7-zip package i would prefer if you will provide just libflzma2-x64.dll which one can drop into any 7z.dll-based program and immediately get access to your codec without sacrificing the original lzma2

If you don't know how to arrange that - i can help

Or merge it with milky's 7z-zstd which already includes several alternative algorithms. I guess/hope that it will opaque the original 7z in due time.
Maybe publish it as LZMA3 or something like that. Perhaps change ari to ANS so it won't remain compatible with LZMA2 and boost a little more the speed?

[Conor]

Maybe the threads can be merged. Not sure if the forum software supports that.

It will be added to 7-Zip-zstd. We already tried to add Radyx but there were compilation problems due to use of C++14. I gave the codec a unique id and added code to replace it with the LZMA2 id when writing, so the resulting .7z file can be decoded with mainline. After some more testing it should be okay to add it to milky's code.

Right now I'm adding progress metering to the library because updates in 7z are granular when using a large dict.

[Bulat Ziganshin]

Maybe publish it as LZMA3 or something like that.

using names like 7-zip pro, zstd2 and lzma3 is unfair to authors of original algos. I rather prefer anything showing strong relation to original algorithm, yet the difference. In particular, Radyx name tells a lot to me, but not to most of users. Given algorithm strong points, i can suggest something with names multi, par, block, f.e. ParLzma2. I.e. the name should let people know that this algo
1) produces lzma2-compatible output
2) employs all cores in more efficient manner than original algo

Perhaps change ari to ANS so it won't remain compatible with LZMA2 and boost a little more the speed?

lzma employs binary ari coders. In order to replace that with ANS-based coders one need to completely redesign compression format, which may require several months of work and skills available only in few specialists (may be, E.Shelwien?). Switching binary ari to binary ans can't buy us anything

I gave the codec a unique id and added code to replace it with the LZMA2 id when writing, so the resulting .7z file can be decoded with mainline.

do you mean it's already implemented in the provided executable or will be in future versions? my english is weak..

After some more testing it should be okay to add it to milky's code.

7z-zs is great project, but i insist that it will be great to publish your codec as separate dll, that can be dropped into other programs too. I think, the main reason behind 7z-zs is GUI support for choosing compression parameters of new codecs, but i think this problem should be solved other way - by establishing an API that allows codecs to expose their parameters to GUIs.

Aniskin already developed first version of such API. If codecs included in 7z-zs and 7z-zs GUI will adopt such API, it will open the way to develop GUI-controlled codecs and codec-controlling GUIs independently instead of packaging both into a single monolithic product

[Conor]

do you mean it's already implemented in the provided executable or will be in future versions? my english is weak.

It's not implemented that way in the provided exe. It was in my 7z-zstd dev branch which was never released, and will be in the new one.

If I get a new codec id from Igor Pavlov and make a drop-in dll, I prefer to make an improved algorithm, e.g. Charles Bloom found that literal encoding is better if always done in match mode. Also some format changes to match it better with the block algorithm. I agree the algorithm should have a unique name, like RLZMA or similar.

[Bulat Ziganshin]

I somewhat lost. So
1) your current codec always generate data compatible with lzma2 format? Or it depends on some parameters?
2) your current codec reuses the lzma2 codec id? i.e. programs using your codec cannot use original lzma2 anymore?
3) what is the way, if any, 7-zip proposes for codecs that have original compression algo, but can be decompressed by existing (standard) codecs?

One more possibility is to extend lzma2 with your match finder, so user can choose bt4/hc4 for original algos, and Radyx MF for a new algo. It's what i have done in my freearc which adds HT4 mf to lzma

Sp please elaborate a bit of lzma2<->rlzma interaction, existing and planned, so we can understand how your codec can be used and how it can't

[Conor]

I somewhat lost. So
1) your current codec always generate data compatible with lzma2 format? Or it depends on some parameters?

Always compatible.

2) your current codec reuses the lzma2 codec id? i.e. programs using your codec cannot use original lzma2 anymore?

Correct unless the workaround is used, but that's not a plug-in dll. It needs a mod to the .7z database writing stage.

3) what is the way, if any, 7-zip proposes for codecs that have original compression algo, but can be decompressed by existing (standard) codecs?

It is not supported because 7-Zip identifies codecs by id after reading the id from the registered info. I had this problem when doing the original 7-Zip-zstd work. I registered the codec with the name Radyx, 7-Zip would read the id 0x21 from the registration info, then when setting up compression it would use the original LZMA2 encoder because it was registered first with 0x21.

One more possibility is to extend lzma2 with your match finder, so user can choose bt4/hc4 for original algos, and Radyx MF for a new algo. It's what i have done in my freearc which adds HT4 mf to lzma

The problem is the matchfinder completes its work before the encoder begins, and multiple encoders run in parallel on the result. Also, rep matches are checked after the match, because if the match table has no match, there's no rep. It works better with a separate encoder.

It will be a while before I try to make a new format.

[zyzzle]

May we please get a 32-bit compile for x86 systems? It is worth trying. Thanks.

[Aniskin]

I registered the codec with the name Radyx, 7-Zip would read the id 0x21 from the registration info, then when setting up compression it would use the original LZMA2 encoder because it was registered first with 0x21.

Did you try to ask Igor about this problem?

[Conor]

May we please get a 32-bit compile for x86 systems? It is worth trying. Thanks.

I've only just compiled an x86 version of the library so will need to do some testing. Maybe in a day or two.

Did you try to ask Igor about this problem?

No but it was pretty clear what was happening. It's unusual to have two ways of making the same type of output that are different enough to need separate implementations.

[Conor]

Attached an updated version to my original post due to an infinite loop bug. Not a library bug but one in the 7-zip encoder object.

[Gonzalo]

Attached an updated version to my original post due to an infinite loop bug. Not a library bug but one in the 7-zip encoder object.

Than you. I have also a different problem in 7zFM. I cannot navigate between folders. It gives me an error saying that the system cannot allocate the right amount of memory, then crashes. Maybe is WINE's fault but I don't have the same issue with vanilla 7z nor zstd-modified

[Conor]

I can't reproduce the problem in Windows 10. Maybe it's something to do with the build options. I'm using the 7z makefiles, which is what 7z-zs uses too but they may have modified them.

[Aniskin]

No but it was pretty clear what was happening. It's unusual to have two ways of making the same type of output that are different enough to need separate implementations.

I asked Igor and he said: Probably it's possible to change that code in 7-Zip. I'll look it later.

[Conor]

Thanks for asking. I found a way to add it as a new matchfinder option into the existing lzma2 compressor class, but it would be neater to make it separate.

I just released a new version with the latest library v0.9.1.

[avitar]

Conor, thanks for this new code.

Just few qs:
0) Be good to integrate with 7z zstd -is this proposed?
1) I've downloaded 64 bit The 7z.exe and 7z.dll clash with those of the 'real' 7zip. If I overwrite these can I still get standard 7z functionality as in 7zip 18.01?
2) Once I've overwritten these can I still use 7z.exe from the standard 18.01 dist or do I need to copy the standard dll?
3) to get max compression do I still use -mx

john

[Conor]

John,

You're welcome

I added the old c++ code from Radyx to 7z-zstd but C++11 build problems meant it was never merged. When the new C library has had some more testing it will be ready to add.

The binaries should go in a separate folder from standard 7z 18.01 because the build didn't include all of the compression codecs and container formats in the full 7-Zip. For the next build I'll look at enabling all of those, and see if I can use the installer too.

The -mx switch should work as before.

[avitar]

Put it in separate folder and tried it compressing a directory from MS Office 2007 799Mbytes. With -mx2 compresses to 679M or so, similar speed to real 7-zip and to arc. rz is a lot slower.

What tells me I'm using the new stuff? Should it just be same compression, but bit eg 5% faster?
john

[Conor]

7-Zip adds its install folder to the search path, so if the modified 7z.exe fails to find the modified 7z.dll, it will find and use the standard one instead. Also if standard 7z.dll is already in memory it will use it. I will look at using a manifest or something to prevent this.

The simplest way to tell is use -mmf=RMF which will error out in the standard dll. Also the -bt option should show under 100Mb RAM usage at level 5.

The lower levels are closer to 7z performance. Level 1 is actually slightly worse in speed vs ratio. The algorithm starts to work really well at level 5 (similar to level 8 in the library dll).

[Aniskin]

I found a way to add it as a new matchfinder option into the existing lzma2 compressor class, but it would be neater to make it separate.

It looks like Igor fixed the problem in 18.02. I created dummy codec with ID=0x21 and now 7zip calls my code during compression.

[Conor]

It looks like Igor fixed the problem in 18.02. I created dummy codec with ID=0x21 and now 7zip calls my code during compression.

Does LZMA2 still work? 7z used to use the first codec that was registered with 0x21, so it's possible that it always uses your dummy codec if it is registered first.

[Aniskin]

Does LZMA2 still work?

Yes. When I select original LZMA2 codec - 7z uses original LZMA2 codec, When I select my dummy 0x21 codec - 7z uses my dummy codec.

[Conor]

That's great. Thanks for letting me know.

[Aniskin]

Is there a chance that you will create a separate codec for 7Zip?

[Conor]

It's the neatest way to add it to 7-zip, so I'll update my 7z fork to 18.02 and modify it.

[avitar]

18.03 out

[Conor]

New release, updated to 18.03 and the radix method is now specified as FLZMA2. It's the default method in the command line exe. Compilation includes Igor's optimized decoder.

https://github.com/conor42/7-Zip-FL2/releases/

[zubzer0]

debug version - need specific dll's

[joerg]

wonderful - thank you very much!