reproductible (compressed) archives

[sebbu]

I'm interrested in reproductible compression :

I have a file (zip or 7z), containing several files.

When I remove a file fom it, add it again, it shouldn't change the file.
When I add a file to an existing archive, or create again from all the (uncompressed) files, it should make the same file.

It's used to share an archive, transmit patchs and ensuring everybody has the same file, no matter if they just got the complete last file or updated it from a previous version.

Here's what I found that works, and can be extracted with any extractor for the original format:

based on zip:
https://sourceforge.net/projects/trrntzip/ torrentzip
http://www.gotwalls.com/files/trrntzip-comment-fix.diff a patch to fix zip files with multiple comments
https://github.com/arogl/trrntzipDN an alternative implementation in .NET

based on 7zip:
https://sourceforge.net/projects/t7z/ torrent7zip

Is there any other/better archive encoder which can give the same guarantees ?
How hard would it be to adapt it to other archivers with compression ?

[Shelwien]

I guess http://nishi.dreamhosters.com/u/shar_v2a.rar because it doesn't include any file attributes
(no compression though, but you can compress the whole archive)

[Bulat Ziganshin]

freearc should be able to create reproducible archives too. we just need to figure out how to suppress all its unnecessary activity
-mt1 - disables unpredictable parallel processing
--recompress - for archive update, ensure that all files are recompressed, in order to make new archive content independent on previous one
--no-dates - drop filetimes from archive, if you need to not depend on them too

of course, you also need to use the same program version and same setting files (such as arc.groups)

[Shelwien]

@Bulat: what about solid blocks or something, to improve diffs?

[Bulat Ziganshin]

what's wrong with them? you can use any solid/sorting/method/arc.groups/-ma settings as far as they are the same, and freearc version is the same.

[Shelwien]

I mean, what would be the best freearc settings for archive sync? Like, update the archive, then distribute the patches with torrent?

[Bulat Ziganshin]

I'm not sure that there is a best solution for everyone. if you compose multiple files into solid block, you improve compression, but increase amount of data that should be sent on update

one of reasons why i developed fa in the first time was my own need to keep large book library and efficiently update it and extract individual book files. so, freearc can be instructed to use 8MB block sizes with -s8m and the it will apply intelligence to updating the archive - i.e. it will keep each solid block in the 6..12 MB range, so f.e. adding one book to the library will require to recompress no more than a single solid block (i.e. 12 MB one became a bit larger and being split to 2 blocks 6 MB each)

It's the smart mode, the program default. For reproducibel archive, you don't need that intelligence, so you should use --recompress option to disable it - this way, archive will be entirely recompressed on every update.

But for reproducible splitting of files into solid blocks there is only one option: -se. It makes a separate solid block for every file extension. We can imagine other staregies of solid block splitting, such as by year, first letter of name and even first digit of its CRC, but freearc doesn't implement them.

Interesting to note, though, that FA'Next becomes much more user-programmable with Lua, and it should be no much problem to add Lua-programmable solid-block splitting strategies. F.e. the compression mode for each solid block is already fully constructed by the Lua code.

I will look into adding that right into 0.12. Thanks for idea. I still look for a ways to employ Lua programmibility for real needs, and that's yet another interesting scenario.