Document Press Clone Post-Mortem

[Krishty]

In the last weeks, I have been working on Compound File Binary Format (CFBF) Optimization – MSI, old DOC/PPT/XLS files, etc. Pretty much what Document Press does – no optimization of file content, just of the container.

tl;dr: I didn’t find an optimal solution. I reached Document Press’ compression rates; sometimes better, sometimes worse. Program & source code at the end of this post. Here’s what I learned:

Previous readings:

1. https://encode.su/threads/336-Document-Press-6-012
2. https://encode.su/threads/897-Docume...-Version-6-013
3. https://encode.su/threads/2864-OLE-O...-for-cmd-tools

---

1. CFBF is a FAT-like file system.

Most of you will know this, but let’s have a clear foundation: CFBF is not compressed but it does support features like concurrent editing, transactions, reversal to earlier snapshots. It’s built on the FAT concept (details in the specification). Compression problems that arise from this:

1. leftover data from earlier snapshots
2. fragmentation

2. There are two CFBF versions.

Version 3 uses 512-B sectors while version 4 uses 4096-B sectors. Therefore, v4 has an advantage with large files. You cannot tell exactly because it depends on the number of files, but the breakeven size is often between 3 and 8 MiB.

V3 files are produced by old MS Office versions and by MSI setups prior to ~2010. MSI setups that have been built after ~2010 are almost exclusively v4 files.


3. Document press never uses the v4 format.

It tries to convert all v4 files to v3. This has huge advantages for small setups (e.g. 72 KiB → 55 KiB) but also great disadvantages for large setups. I have never seen Document Press reducing the size of a v4 CFBF file. It only copies the input because that’s smaller.


4. Defragmentation *is* important.

That’s what Document Press’s -opt does. CFBF allocates small files (<4096 B) and large files (≥4096 B) from different streams, but they share the underlying sector system. Writing small files, then large ones, and again small ones will almost certainly lead to fragmentation. Tom Jebo from the Microsoft Open Specification Support Blog (defunct since two weeks) has written about it in a two-part series (Exploring the Compound File Binary Format, Exploring the Compound File Binary Format (part deux)), but you should not trust the source code because it bloats the resulting file. Document Press got it right as far as I can see.

The true™ way of defragmenting a CFBF file (as far as I tried) is:

1. Create the directory and all streams (with empty sizes!).
2. Copy small files. Or large files? I don’t know, see below. But keep them seperate from each other!
3. Properly close all open handles or else there will be scratch space or snapshots left in the file.

5. Then, it gets really fuzzy.

Once you removed needless snapshots & scratch space from the file, you’ve reached the minimal size. Defragmentation will not reduce size any further, so you have to compare sizes after compression e.g. with LZMA.

And now you’re left with the dilemma of finding the correct way of sorting files to help compression. I didn’t look into this and neither did Document press (I bet it and I got bet randomly).


6. MSI probably uses a bastardized v4 format.

This is something I didn’t know but would like to have information on.

Even though v4 is defined to have 4096-B sectors, I have seen MSIs missing 7/8 of the last sector, i.e. it’s just 512 B instead of 4096. I don’t know why that is. But I certainly know that it defeats optimization because no matter how hard you defragment, your standards-compliant result will always be 3584 B larger than the fragmented version you started with!

Moreover, I have seen many (but not all!) v4 MSIs being already optimized and defragmented, including some I have generated myself wia WiX or via MSI API. Considering the last-sector trick, it’s impossible to beat them. (Meaning Microsoft has put some effort into this after ca. 2010, which is good, isn’t it?)

---

That’s it. You can download my program here (Windows x86-32): https://papas-best.com/downloads/bes...6/bestcfbf.exe
And the C++ source code here: https://papas-best.com/downloads/bes...e/bestcfbf.cpp

If you want to optimize and defrag a CFBF file, run it with

bestcfbf <in> <out> [-v4]

For normal files, I recommend using it twice (one time with -v4 and one time without) and picking the smaller file. If it’s larger than the input, then you probably have a v4 MSI.
If you prepare the CFBF for compression, also use Document Press on it and select the one that compresses best.

I think the last way to improve anything would be reading and writing individual FAT sectors (which I didn’t, I just used the Shell Lightweight COM API). Igor Pavlov is a skilled programmer and there’s some chance that Document Press already does that.

[Krishty]

P.S. – Small benchmark (all sizes in KiB):

                       original      bestcfbf   bestcfbf -v4  Document Press 6.01
RSS Bandit setup        10687.5         10687          10656                10687
Word 2000 sample file        70            64             88                   64
STL Viewer setup             76            55             72                   55
Movie Quiz.xls           3650.5          3650           3644                 3650

As you can see, it performs like Document Press on small files and outperforms it on large files with -v4. I should probably add another test with large, post-2010 MSI files where neither gains anything.

[maadjordan]

its a good start. Document press also support CHM files.. can you add it?

[Krishty]

I’m afraid not – CHM cannot be opened through the compound file API I use, and the fact that 7-Zip opens them flawlessly makes me think that Igor Pavlov coded a special solution for it …

[Shelwien]

7-zip actually has its own parsers both for compound format and for chm/lit, they're different:
https://github.com/kornelski/7z/tree...ip/Archive/Chm
https://github.com/kornelski/7z/blob...ndler.cpp#L875

[maadjordan]

CHM and compound files are different.

what i was thinking is that calling chm compaction is a system dependent call built into windows but i could not confirm this. I asked Igor to issue document press source code but it confirmed that the source code was lost with a hdd failure.

Same thing for keytools (calling a system register) which can let chm files reach LZX18 (instead of default LZX16 and document press LZX17 but spec allows for LZX21 none can reach).

may be we can de-compile both programs to confirm that.

[Krishty]

To build my MSIs, I use the Cabinet API which is built into Windows and supports LZX21.

With deep knowledge of the CHM format, one may be able to use that API to extract existing data and re-pack it using LZX21, then re-insert that into the CHM and fix up everything around.

I have never had a look at the CHM format, though. I have also not seen references to CABINET.DLL in Document Press 6.01, so its LZX magic likely was hand-written by Igor.

[Krishty]

Just a note: I investigated whether this works with digital signatures, and indeed it does.

Digital signatures in compound files are generated from the checksum of the content – not the container – so any digital signatures on MSIs or DOCs or whatever are still valid after optimization.

[maadjordan]

just for the record and later usage .. there was a gui program that can edit and defrag such files called eDoc

this is the wayback link

https://web.archive.org/web/20021008...doc/index.html

[Krishty]

There are two small glitches with the current version of CFBF Optimizer.

It’s time stamps. CFBF does not store one time stamp, but rather a dozen of it. The time stamps from the summary fields – these are what you see in File Explorer’s Details tab and with standard tools – are copied fine. But mind that CFBF files describe an entire file system, and the so-called Root Entry and Object Pool in the system come with their own modification dates. These are set to the current system time during optimization.

I don’t know a program that actually uses them, so they are only of forensic interest, if any. But one has every right to not call this lossless.

Root Entry and Object Pool are standardized objects in the format, but I haven’t quite figured out the API to access them, so there’s no quick magic fix I could provide immediately.

I’ll keep you updated.