Shared Brotli Format

[Jyrki Alakuijala]

We are preparing an extension of Brotli into full blown shared dictionaries. The shared dictionaries include LZ77ish dictionary model (similar to those that have been supported by Brotli repository — but not RFC7932 use — and ZStdandard for a long time), but also the Brotli's (distance,length) pair mapping dictionary mechanism and Brotli's word transforms to get variations of the dictionary -- for example for plurals, articles, inflection, prepositions, and capitalization in human spoken languages. In addition to what we have in Brotli, we include a context model that allows the dictionary to be reordered based on the two last bytes in the stream.

There are a few other things in the scope of Shared Brotli than just shared dictionaries. It includes a large window extension of brotli, a patching mode that is faster and more dense than Bsdiff+Brotli, and a framing format.

Shared Brotli is described in https://datatracker.ietf.org/doc/dra...brotli-format/

In HTTP content encoding simulations of Shared Brotli we have often seen a 40–60 % reduction of data transfer in comparison to the traditional RFC7932 Brotli — after the shared dictionary has been transferred.

[SolidComp]

Does the Brotli dictionary word transforms feature support wrapping words in spaces? What about punctuation like periods? We often see people construct dictionaries with words like: the, they, and, or, etc.

Those terms are wrong. the should actually be a five-character string [space]the[space]. That is, if it's the lowercase the. The uppercase The should actually be in the dictionary as [period][space]The[space]. That's a six character string that works for all sentences except the start of a paragraph. they, and, or and lots of other lowercase non-sentence-terminating words should be wrapped in spaces in the dictionary. Capitalized forms of Therefore, Moreover, Thus, etc. should include the closing period of the sentence that just ended, the following space, then the word, then another space, since that is how they will be situated in common and correct English. (Obviously there will be typos where people don't have a space after a sentence or something, but so what? It won't be the majority case.)

[Jyrki Alakuijala]

Those terms are wrong.

Shared Brotli will allow you to do this in the shared dictionary. You can define transforms with your own postfixes and prefixes. Normal Brotli is what it is. You can see its transforms in rfc7932, near the end.