demo of lzma file alignment effect

[Shelwien]

http://nishi.dreamhosters.com/u/lzma_align_v0.rar

file1   262151
file1a       9
file2   786432

1.7z    827248 // 7z a 1 file1 file2
2.7z    809983 // 7z a 2 file1 file1a file2

It might be even better if lc2 (lzma parameter) is used.

In short, lzma is weird like that, so all files in solid archives have to be padded to filesize%32==0

[Piotr Tarsa]

have to be

but how they can be?

[Shelwien]

> but how they can be?

By padding them like I demonstrated?
Or just by taking it into account in a new format implementation?
(The padding doesn't have to be actually stored anywhere, its just needed to fix up the data position in the codec)

[Piotr Tarsa]

without a new format, we will end up with unwanted files on extraction, i think.

[Shelwien]

yes, but for installers and such it doesn't matter.
Also we can use "nul" for names of pad files

[m^2]

Could you explain what's the reason for the effect?
Does it impact lzmarec too?
How came Igor didn't do it by himself?

[Shelwien]

> Could you explain what's the reason for the effect?

lzma doesn't treat the match distances as a single "alphabet".
It only has a model for log2(dist) + high bit and low 4 bits,
while all intermediate bits are stored without compression.

Thus, any changes in the data that affect the distance alignment
would also affect compression.

Of course, my example here is intentionally designed to amplify
the difference (its still not the worst possible case though),
but the same effect is also noticeable with "natural" files.

> Does it impact lzmarec too?

Yes, lzmarec uses a very similar distance decomposition:

1.7z.rec 755038
2.7z.rec 737363

> How came Igor didn't do it by himself?

lzma is basically the only codec (besides its clones) which has such a quirk,
so maybe he simply didn't think about this.

But its also a fact that there're many filetypes for which
lzma compression would be affected by alignment of their
start position (eg. exes), and that all existing archive formats
with lzma support and solid compression don't fix up
the stream position on transition between files.

Anyway, there're many possible improvements for lzma which are not
currently implemented (eg. http://encode.su/threads/1546-init-f...terals-in-lzma )
so its nothing special.