ms cab recompression

[Urist McComp]

Edit 2018-10-24
Uploaded new binary:
- Some bug fixes, should be a lot more stable now
- "light" support for cabs that mix lxz and other compression types: the unsupported types are now simply copied instead of crashing the program.
- readme file

Edit 2018-06-06:
Uploaded new binary, no new features, just compiled differently to be faster and to include most of the dependencies.

Hi,

I just finished a first experimental version of a ms cab recompressor. It only has some limited functionality but maybe it is already useful for someone. Althoug you should not trust on it being reliable or stable, though.

I planned to do it somewhat different at first, but then I decided on using the ms cab libs to easily achieve small recompression info size for most of the files.

- It handles files with lzx compression only.
- It is pretty slow because it is a debug build for keeping the assertions. And it misses a lot of optimizations
- x86-64
- Windows only, sorry (uses windows libraries for compression).


TODOs (unordered):
- support for mszip
- support for quantum compression
- support for multi file archives
- probably bugfixes
- recursion
- scanning for cab signatures in bigger files (tar for example)
- Maybe linux support. Will probably increase recompression info file size, though.
- Efficient handling of files compressed by third-party cab/lzx compressors
- smaller recomp info size?
- your suggestion here

I am pretty unsure about the decompression output. Right now it extracts all recompressible files to a given directory so their file type can be detected easily for further (re)compression. But maybe something else would be more useful?


Regards

[Stephan Busch]

How can the original .cab be restored once the extraction has taken place?

[Urist McComp]

Just run it with a slightly different command line: -r instead of -d and a name for the .cab file that doesn't already exists (it should not overwrite existing files).


From the included "help":
Decompression:
-d archive.cab archive_recomp_info.cab-ish decompressed_files_dir
Recompression:
-r recompressed_archive.cab archive_recomp_info.cab-ish decompressed_files_dir

Edit:
The names of the files in the decompressed_files_dir need to be unchanged for it to work. And there might be issues with utf8-names (haven't tested).

Edit2:
I just realized I might be using the term recompression incorrectly.
So here is the basic workflow:
1) decompress cab using my tool
2) compress the output any way you like
3) decompress your compressed data again
4) restore the cab using my tool

I assumed step 4) to be called recompression, although calling 2) recompression seems plausible, too. So how would you call these steps?

[nikkho]

Very interesting. I would like to add it to FileOptimizer tool chain when ready!

[Gonzalo]

Tell you something? I've been day-dreaming about lzx recompression for years! Thank you very much, Urist!


A few comments about your TODOs and questions:


1) A recompressor is only good in company of other tools. Perhaps you should consider integrating your tool inside Precomp or Fairytale, given they are doing exactly the same as your program but for all the other formats. In doing so, a single tool could perform the whole recompression (recursion included) for all the data inside a .cab file, including jpg, png and gif images, for example.


2) I don't think quantum was ever even thought of being used in production. I doubt you'll find any real life samples somewhere to play with so maybe you should channel your skills to some other part of the program.


3) If you plan to make it multi-platform (which IMHO is a must) perhaps you can take a look at this tool. It's very mature software ever improving. It supports most Microsoft compression algorithms and will compile everywhere.
Quoting its author:

wimlib implements the XPRESS (Huffman variant), LZX, and LZMS compression algorithms. All these compressors provide slightly better compression ratios than their Microsoft equivalents. However, I've only implemented the chunk-based compression that is needed for the WIM file format; there is no "sliding window" support. So the LZX compressor would require modifications to be usable for CAB recompression, since (I think) CAB files use LZX in "sliding window" mode. Though of course, if you actually want to implement LZX CAB recompression, it would be much easier to start with wimlib's LZX compressor than to start from scratch.

4) For <cab inside a container> case, again, look at Fairytale. Its very core is based of data recognition so it is designed to easily add new parsers and automatically make them benefit from and to other parsers and parts of the compression process, like streams de-duplication.

5) About this:

1) decompress cab using my tool
2) compress the output any way you like
3) decompress your compressed data again
4) restore the cab using my tool

I assumed step 4) to be called recompression, although calling 2) recompression seems plausible, too. So how would you call these steps?

I would call "pre-compressor" a program that can do your (1) and (4) operations losslessly (example: Shelwien's reflate and unpackmp2). And a program capable of doing 1 through 4 is a re-compressor. For example, packJPG, packMP3, paq8px.

Keep up the good work, man! This is ground-breaking stuff

[Stephan Busch]

Dear Urist McComp,

welcome to the show.
I also have been dreaming about lzx recompression for years.
It's a pleasure to see that this dream has finally come true.
What would you think about adding your recompressor to Precomp just like deus-libri did it with Preflate?
This would be a great addition and it would probably save you time.

[Urist McComp]

nikkho, Gonzalo and Stephan, I made a "remix" of your posts below to avoid some redundancy (at least we are talking about compression, right? ), I hope i didn't miss anything.

Very interesting. I would like to add it to FileOptimizer tool chain when ready!

Feel free to do so!
Although I am not sure if my tool has the same scope as File Optimizer (correct me if I'm wrong): File Optimizer does (for example) png->smaller png. My tool (combined with a compression tool of your liking) on the other hand does cab -> compressed cab -> original cab.

Maybe you could add a short description on your sf-page ( https://sourceforge.net/projects/nikkhokkho/ ) to explain, what File Optimizer actually does, something like: "compresses already compressed files further without changing the file format."

Also It will probably take me quite a while to develop a reasonably stable version.

Tell you something? I've been day-dreaming about lzx recompression for years! Thank you very much, Urist!

I also have been dreaming about lzx recompression for years.
It's a pleasure to see that this dream has finally come true.

Happy to help making your dreams become reality

1) A recompressor is only good in company of other tools. Perhaps you should consider integrating your tool inside Precomp or Fairytale, given they are doing exactly the same as your program but for all the other formats. In doing so, a single tool could perform the whole recompression (recursion included) for all the data inside a .cab file, including jpg, png and gif images, for example.

3) If you plan to make it multi-platform (which IMHO is a must) perhaps you can take a look at this tool. It's very mature software ever improving. It supports most Microsoft compression algorithms and will compile everywhere.

4) For <cab inside a container> case, again, look at Fairytale. Its very core is based of data recognition so it is designed to easily add new parsers and automatically make them benefit from and to other parsers and parts of the compression process, like streams de-duplication.

What would you think about adding your recompressor to Precomp just like deus-libri did it with Preflate?
This would be a great addition and it would probably save you time.

1) Sounds interesting.
Although 1), 3) and 4) require me to solve the problem of replacing the ms compression libraries because I need to generate bit-identical cabs. I have plans to do so (now that I followed your hints I think my plans are similar to what preflate does), but it will pretty sure increase the amount of additional information required for restoring the orignal cab.

I haven't found much about Fairytale except for https://encode.su/threads/2927-Fairytale and the links included there. Is there more information about it's current state? Is it somehow (planned to be) a successor to precomp with a lot more features?
Edit: Ok, it is not. The name of the repository got me thinking it was.

If I understand it correctly Precomp is used as a "frontend" for preflate (zlib stream detection), I still would need to do cab/lzx stream detection by myself. But maybe it still can help in some way. I'll just have to take a look.

2) I don't think quantum was ever even thought of being used in production. I doubt you'll find any real life samples somewhere to play with so maybe you should channel your skills to some other part of the program.

Thanks for the info, I think I already saw it, but I am not 100% sure. I didn't do any thorough research.

Keep up the good work, man! This is ground-breaking stuff

welcome to the show.

Thanks - Although I'm not sure about the ground-breaking part (at least concerning the techniques I used).

[nikkho]

Maybe you could add a short description on your sf-page ( https://sourceforge.net/projects/nikkhokkho/ ) to explain, what File Optimizer actually does, something like: "compresses already compressed files further without changing the file format."

Thank makes sense. I have clarified it here: https://nikkhokkho.sourceforge.io/st...=FileOptimizer
As for project page, it contains other projects by me, not only FileOptimizer, so I am not sure.

[maadjordan]

Many thanks to initiate it.. I you will allow me to add some todo list with the following features:
A- MSCAB:
- changing mszip and quantum methods to LZX (it will work as cab optimizer).
- improving LZX compression by using "wimlib" LZX routine as its faster and makes smaller files. (again cab optimizer).
- expose internal files for other programs (if a png file inside cab file to be extracted and use png optimizer to reduce size prior repack into cab again)

B- Adding CHM files
- https://github.com/mdsteele/rust-msi

C- Adding MSI files
- https://github.com/mdsteele/rust-msi

D- Adding WIM files (its not a must as wimlib can handle/optimize/recompress wim files)
- https://github.com/jhermsmeier/node-wim

E- LZX presence in other containers
- https://en.wikipedia.org/wiki/LZX_(algorithm)

D- Useful packages:
- the libmspack will be very helpful to use https://github.com/kyz/libmspack
- for portability https://github.com/dorkbox/CabParser
- wimlib from https://wimlib.net/
- lzx streams handler in rust https://github.com/mdsteele/rust-lzxd
- https://github.com/jhermsmeier/node-lzx
- https://github.com/jhermsmeier/node-cabarc
- https://github.com/coderforlife/ms-compress

[Stephan Busch]

Dear Urist,

Fairytale is still in the planning stage and it is meant to be a new archiver with many interesting technologies including Precomp, Preflate and CLZ.
Precomp is used as frontend for Preflate, and your .cab recompressor would also be nice to have in Precomp because it fits well
and is a great enrichment to the existing portfolio, and you would get help from the community and authors.
Like Maadjordan said, there are some helpful packages around and I would also vote for support of .msi and maybe .chm and .wim file formats.

[Gonzalo]

Well, I stand corrected before anything. I did found one quantum-compressed cabinet yesterday. Just one amongst the hundreds I checked, and tiny, but it does exist! I attached it to the post because I don't want it to feel no love in the world.


Precomp is way older than preflate and also does recompression without it but only to deflate streams created with the zlib library. Preflate takes the matter a little further by making possible to recompress all other deflate streams, no matter what code they were created with. There is a little drawback, some overhead that needs to be added. And yes, precomp acts as a frontend for preflate.


Fairytale will be a full-featured file archiver and compressor (think 7-zip on steroids) The idea is to make a one-fits-all solution, ranging from quick on-the-fly archiving to record-breaking ratios, with an emphasis on efficiency. To do that, it will use several state of the art libraries (probably preflate will be one of them), several tricks learned from previous projects and its own file format, which is already designed on paper and half-way implemented. Precompression finds its spot on the high ratio end of the speed / strength trade-off.
I'm writing some documentation for the project but I won't release it until we have something 'usable'. As of today, almost all infrastructure is in place and there are very promising reports but we don't have a program that 'does something' yet. Nevertheless, I think you'll find useful to read what it is supposed to do, to where the developers are going, so I'll p.m. to you what I have now.

[Urist McComp]

I'll refer to changing the compression inside a cab but it still staying a cab WHITHOUT the possibility to recover the original file as optimization. Decompressing the cab for compressing it with other methods and then restoring the original cab I will call recompression. Are these the correct terms to use?

Thank makes sense. I have clarified it here: https://nikkhokkho.sourceforge.io/st...=FileOptimizer
As for project page, it contains other projects by me, not only FileOptimizer, so I am not sure.

Ok, then I misunderstood your sf-page, especially because of the features-list. Thanks for the clarification.

Many thanks to initiate it.. I you will allow me to add some todo list with the following features:
A- MSCAB:
- changing mszip and quantum methods to LZX (it will work as cab optimizer).
- improving LZX compression by using "wimlib" LZX routine as its faster and makes smaller files. (again cab optimizer).

I didn't think there was such a high interest in optimizing cab files. You are the 2nd person to ask for it, so I will add it to my TODO list but for now with pretty low priority.

- expose internal files for other programs (if a png file inside cab file to be extracted and use png optimizer to reduce size prior repack into cab again)

I am not 100% sure if I understand you correctly, because this is already the intended use of my tool. Just compress the decompressed files any way you like. For restoring the original cab just decompress and then restore the cab using my tool.

B- Adding CHM files
- https://github.com/mdsteele/rust-msi

Thanks, I already had this somewhere in my mind, but now I know a little more and I hope there will be some good gain possible for recompression because of the small block sizes. On the other hand .chm files should normally not be that large and slowly become less frequent I guess.
Are you sure this is the link you wanted to provide, though? At the first look I only found information about .msi files but maybe I need to do some more thorough reading.

C- Adding MSI files
- https://github.com/mdsteele/rust-msi

Thanks, this could be interesting with msi files being pretty common.

D- Adding WIM files (its not a must as wimlib can handle/optimize/recompress wim files)
- https://github.com/jhermsmeier/node-wim

I think wim files are pretty interesting for recompression, I'm already curious about the possible gains, especially when using recursion. Already had this at the back of my head.

E- LZX presence in other containers
- https://en.wikipedia.org/wiki/LZX_(algorithm)

I don't even know where to get files of the additional formats listed there

D- Useful packages:
- the libmspack will be very helpful to use https://github.com/kyz/libmspack
- for portability https://github.com/dorkbox/CabParser
- wimlib from https://wimlib.net/
- lzx streams handler in rust https://github.com/mdsteele/rust-lzxd
- https://github.com/jhermsmeier/node-lzx
- https://github.com/jhermsmeier/node-cabarc
- https://github.com/coderforlife/ms-compress

Thanks, although I already do decompression by myself. Compression for recompression is another issue and at least for cab files created by the ms-libraries using them again is the best solution (for smallest restore info size). Sadly this is not portable (more on my planned solution below). These librariles may be interesting for recompressing files created by them, though. Maybe they also will help me spot some bugs in my code, too. Or help me make it faster.



My plans for portable and universal lzx (and cab) reconstruction:

As of now I am using the ms cab libraries for reconstucting the cab, so I only need very few additional information for files created using the ms libraries (which should be the majority of all cab/lzx files out there). Sadly this is not portable and will not work for files compressed by alternative implementations. For the latter I will probably try to include as many of the other librariesas possible (as long as there are no license issues, hopefully there will be none after open-sourcing my tool). But then there still is the issue with the ms library compressed cabs.

For this I plan the following (some more complex contex modelling left out of the description):

A matchfinder to find ALL match candidates. I already implemented an unoptimized one a while back with about 240N memory usage - which is slightly more than I prefer... It had a few special properties which will come in handy, though. So I am planning on reimplementing it with about 8N. Once I got the longest match candidate I store the difference in length (to the real match). After that I will get a list of all possible matches with the real match length, ordered by shortest to longest distance and store the index of the one that is really used. I will also need some modeling for optimal parsing and literals that are encoded instead of a possible match (probably mostly due to parsing optimization). I will also either use the information in the original huffman tables or try to compress them as much as possible using the information given by the actual matches.
Maybe I am reinventing the wheel here? But I probably will do it anyway for learning/fun. And also because I want to reuse the matchfinder to improve on some promising tests I already did, but this is another topic. It may also be useful for recompression of other formats, but as I am probably not the first one to do this, it will not be such a big thing.

Once I got this done and thus have a cross-platform version I plan to integrate it into some other tool, maybe precomp, maybe fairytale. Once the time comes I'll just have to see which one fits better/is easier to integrate into/is more useful to users.

[Heinrich]

It can be made open source?

[Urist McComp]

It can be made open source?

That is planned for the future, but I want to do several improvements, first. Why are you asking, do you have any special plans for it?

[Heinrich]

Why are you asking, do you have any special plans for it?

I have some skills in C++, C# and a great desire to re-archive the installation files of programs, but not enough time. So maybe I could help something.

[Urist McComp]

I have some skills in C++, C# and a great desire to re-archive the installation files of programs, but not enough time. So maybe I could help something.

Thanks for the offer. I like getting hints on how to improve my code , sadly I usually don't have the opportunity to get hints.
Or maybe there even is something you'd like to implement. You are probably most interested in *.msi handling?
Although right now I think I prefer implementing most of the code myself.

One issue that is keeping me from releasing the source right away is selecting a proper license. Or even multiple ones.
- I'd probably like to use something with a copyleft.
- Maybe also something less restrictive?
- Can I even add another license later if someone else has already contributed? Do I need their consent? Does this maybe depend on the license?

It's way more fun to do some actual programming, so I have avoided this topic until now. I guess I have to do some reading. You don't need to comment on my list with anything that I can easily find by googling, use your time for something better . But if you have some personal preference or something I'd like to hear about it.

Another thing:
Has anyone already tested my tool? Does it work for you on pure lzx cabs or are there some files it fails to handle?

[Gonzalo]

=double post=

[Gonzalo]

It fails on everything for me. I tried to use it on several files. It can be WINE's fault, though. I was waiting for the next release to see if anything changed

[Urist McComp]

Fails as in creates a new file that is a few bytes bigger than the old one? That is expected using wine as its cabinet.dll replacement creates slightly different datastreams as the original one. Maybe you can provide the original dll to wine, then it should work.
Recreating the original cab should still be possible, though (as long as the same dll is used for both steps).

Or do you mean fails as in crashes? This should not happen for lzx cabs. Then these files would be interesting for me.

[Stephan Busch]

I haven't had time to test it so far, but will do so tomorrow

[Gonzalo]

Fails as in creates a new file that is a few bytes bigger than the old one? That is expected using wine as its cabinet.dll replacement creates slightly different datastreams as the original one. Maybe you can provide the original dll to wine, then it should work.
Recreating the original cab should still be possible, though (as long as the same dll is used for both steps).

Or do you mean fails as in crashes? This should not happen for lzx cabs. Then these files would be interesting for me.

Nope. Fails as in it crashes every single time. No particular file, but all of them. And yes, they are compressed with lzx. Which binaries should I replace to rule out WINE as the culprit? Only cabinet.dll?

[Stephan Busch]

Dear Urist,

I cannot find the dlls required to run the executable.
Visual Studio 2015 runtimes are installed on my Win10.
Can you please provide the dlls needed to run the exe?

[Urist McComp]

Ok, I updated my first post with a new binary. Maybe it crashed under wine previously because I used the new c++17 filesystem features. Providing the original cabinet.dll by microsoft ist probably still needed to be able to recreate archives compressed with it. I'm still working on replacing it but it takes some time.

[Gonzalo]

This is from the newest binary (2018-06-06), with cabinet.dll replaced:

Starting to decompress cab for recompression. This may take a while.004a:fixme:cabinet:FCIAddFile compression 1203 not supported, defaulting to none
Assertion failed: 15 <= window_shift && window_shift <= 21, file ..\..\..\src\lib\cab\cab_folder.cpp, line 26
004a:fixme:kernelbase:AppPolicyGetShowDeveloperDiagnostic 0xfffffffffffffffa, 0x2343e8
004a:fixme:kernelbase:AppPolicyGetWindowingModel 0xfffffffffffffffa, 0x2343a8

[Urist McComp]

Thanks for testing and your report.
It looks to me like this still was the cabinet.dll from wine (the fixme part is not my code). At least if the "1203"-compression type is a hex value it should be valid. It looks like it could be a pretty small file.

The assertion on the other hand is in my code. Maybe your cab uses features that I still don't support so I read from a wrong position.

I attached one small file that definitely works for me, just to make sure whether the problem is wine-reladed.

If it still doesn't work I'll probably have to setup wine myself.

[Gonzalo]

I believe WINE only thinks is its own dll so it consequently tries to debug it. After all, I'm not supposed to mess around with system files. I just looked up the file and overwrote it, instead of using the 'override' tool from WINE.
Anyway, this is my output for your "cabtest.cab":

wine recomp_ms_cab -d cabtest.cab cabtest.cab_ish cabtest000b:fixme:winediag:start_process Wine Staging 3.9 is a testing version containing experimental patches.
000b:fixme:winediag:start_process Please mention your exact version when filing bug reports on winehq.org.
Starting to decompress cab for recompression. This may take a while.
002f:fixme:cabinet:FCIAddFile compression 1203 not supported, defaulting to none
Assertion failed: 15 <= window_shift && window_shift <= 21, file ..\..\..\src\lib\cab\cab_folder.cpp, line 26
002f:fixme:kernelbase:AppPolicyGetShowDeveloperDiagnostic 0xfffffffffffffffa, 0x2343e8
002f:fixme:kernelbase:AppPolicyGetWindowingModel 0xfffffffffffffffa, 0x2343a8

I might add, no files are extracted either. Not only once of all the times I tried. Being that the first anomaly I saw, maybe is the root of the other issues...

[Urist McComp]

Thanks for your tests, so I probably have to set wine up myself for further debugging.

I might add, no files are extracted either. Not only once of all the times I tried. Being that the first anomaly I saw, maybe is the root of the other issues...

This is expected beahavior: The files are decompressed to ram with my own code without any temp files. After that I pass them to cabinet.dll for compression. Only if they can be compressed bit-identical again the decompressed files get written to disk. Otherwise they are stored in their old compressed form.

So when reproducing the compressed form fails (as it seems to be the case) no decompressed files will be written. On the other hand if that was the only thing failing, a recompression info file with a few bytes more than the original cab would be stored.

[Stephan Busch]

Dear Urist,

recomp_ms_cab fails on this LZX file: https://drive.google.com/file/d/1vuu...ew?usp=sharing
cabinet.dll was not in the same directory during the test run.

[Urist McComp]

Hi Stephan,

sorry for my late reply, I was on vacation. Thanks for your report, I can confirm something is not working, I still have to analyze what exactly, though.

[Urist McComp]

Uploaded a new version:
- Some bug fixes, should be a lot more stable now
- "light" support for cabs that mix lxz and other compression types: the unsupported types are now simply copied instead of crashing the program.
- readme file