What Compressor Used?

[brispuss]

Trying to determine what specific compressor was used in compressing the file images.dta?

This file is part of the game "Nightmare Adventures - The Witch's Prison". Downloadable from here -

https://www.bigfishgames.com/games/5962/nightmare-adventures-the-witchs-prison/?pc

The file images.dta is 212 190 501 bytes in size.

The file seems to extract OK using 7zip. But on re-compressing file contents (as a zip file), the compressed file size does not match the original file size, despite trying different compression settings! The new compressed file size is usually smaller than the original. This is of some concern, as ideally the original and re-compressed file sizes should match. Although I understand a (slightly) smaller file size might still work, but this is dependent on the game software and how it reads this file.

Using a hex editor, the file header reads -



This suggests the images.dta file was compressed using a compressor using (a variant of the) "zip" algorithm(?)

Can anyone please confirm which specific compressor was used, and verify that re-compressed data (using the same compressor?) is the same as the original images.dta file?

Thank you!

[Shelwien]

7-zip has a custom size-optimized algorithm for zip compression - better/slower than zlib, which is normally used for this.

You can either process this file with precomp - which is a tool specifically made for a task like this.
Or you can extract it, then try to compress with gnu zip - you can find it as zip.exe in UnxUtils.zip: https://sourceforge.net/projects/unx...utils/current/
zip.exe -r -1 arch1 *
...
zip.exe -r -9 arch9 *

Then use a binary diff utility to see which zip level matches the data better (since some zip headers would be different in any case).

[brispuss]

Thanks for the comments!

I must be missing something(?) But how does precomp determine which compressor was used to create the file images.dta?

I tried the zip.exe from the link, but this compressor doesn't produce the right file size despite trying different compression settings.

The idea is to archive various programs (games and utilities), but to compress them as small as possible before archiving. Part of this process may require compressing parts of software which are already compressed as well.

This is the case here for this particular game where the images.dta file is compressed. It is wanted to decompress this file. Then compress the file contents using precomp and/or other compressors to higher compression ratio. Then compress the entire game for archiving.

Later, software (game) is de-compressed. Then the further compressed parts of the software are also de-compressed. Then, ideally, the de-compressed parts are re-compressed using the compressor which which used to compress software parts originally (by the developer). In this case here, the file images.dta must be re-compressed using the same original compressor (OR using another compressor, provided the file size output is the same and the content is the same as the original images.dta file).

?

[Shelwien]

> I must be missing something(?)
> But how does precomp determine which compressor was used to create the file images.dta?

It doesn't.
Previously precomp just looked for zlib options to match the detected zlib streams...
like I described with zip.exe, but automatically.
Now (v4.7) it uses this library: https://github.com/deus-libri/preflate
Which in addition to actual content, which can be unpacked from a deflate stream,
also additionally stores encoder choices that affect the compression output.

> I tried the zip.exe from the link, but this compressor doesn't produce the
> right file size despite trying different compression settings.

As I said, you'd have to use diff on created zips and original file.
Aside from actual compressed data, there're also multiple ways to write the zip headers -
zlib is the library most frequently used for zip compression, but there're
much more libraries to create the headers.

https://github.com/sisong/HDiffPatch/releases

> The idea is to archive various programs (games and utilities), but to
> compress them as small as possible before archiving. Part of this process
> may require compressing parts of software which are already compressed as well.

Unfortunately you are not the first to invent this.
This is called "repacking" and is quite popular.
There're also many tools created specifically for this.
For example: http://aluigi.altervista.org/quickbms.htm

> In this case here, the file images.dta must be re-compressed using the same
> original compressor (OR using another compressor, provided the file size
> output is the same and the content is the same as the original images.dta file).

Yes, but its not very simple.
Basically a recompression tool has to be written for every codec specifically.

Then, container headers can be easier to reproduce, but they are usually
different in every game, so this requires some programming.

Or you can use some binary diff tool to patch the recompressed file
into actual game file.

[schnaader]

The file images.dta is 212 190 501 bytes in size.

The file seems to extract OK using 7zip. But on re-compressing file contents (as a zip file), the compressed file size does not match the original file size, despite trying different compression settings!

The file is a ZIP archive indeed, but everything is stored (because the PNG files wouldn't compress much). You get a very similar file when using the option "ZIP" and "Store" in 7-Zip:

212.190.501 images.dta
212.265.025 images.zip

[brispuss]

I can see there are some difficulties here!

Apart from decompressing many compressed file formats, Quickbms can only update existing compressed files, it can't (re)create (new) compressed files. So this utility is not entirely useful in my case, since I need compressors that must be able to recreate original files (exactly) without the original files being present.

The intention is to archive software. After archiving, the original software will be deleted.

When decompressing archived software, the decompression and any re-compression must be such that it reproduces that same content as per the original content without having to compare the files to original files (which are no longer available as they had been deleted to save HDD space).

So trying to use a binary difference patching utility is not going to work unless the original files are re-downloaded to use as a comparison to the decompressed/re-compressed files. This is not really the best option, as I'm trying to avoid extra steps in "re-creating" original software.

[Shelwien]

Maybe you misunderstood something, but what I suggested is this:
1) you have 1.zip and want to recompress it
2) so you do something, but end up with 2.zip!=1.zip
3) in this case, a patch can be created, like diff 2.zip 1.zip 2-to-1.patch
4) so in your installer, you'd have the data necessary to make 2.zip and 2-to-1.patch,
and after 2.zip is created, you can patch it back to 1.zip

Yes, its a somewhat ugly method, but compression-wise its usually ok, if the patch has a reasonable size.

Otherwise you'd have to reverse-engineer and write tools for working with specific game formats,
basically new tools for every game.

> Quickbms can only update existing compressed files

Its also not an issue, since you can zero out the place used by extracted files in the container.

[brispuss]

OK. Before deleting the original file and after creating a decompressed test file as a test, THEN do a comparison check between the original and the decompressed test files. For any differences between the two files, use a patching utility to apply a patch to the decompressed test file to make it match the original file.

Any suggestions on a good reliable file patcher?

Thank you.

[Shelwien]

https://github.com/sisong/HDiffPatch/releases as I said.
It also has limited support for deflate recompression already integrated.

[brispuss]

Thanks.

Sorry, I missed the first post for this utility!

Command line based, with five executables to choose from(!?).

Although command syntax is listed when running any of the executables with no parameters, it is not entirely clear which executable I should use to create a patch based on differences between two files?

Frankly I find these executables difficult to understand and use!? Are there no "driving instructions" for these executables?

Are there easier to understand and to use alternative file differential patchers rather than this one? Preferably with GUI?

[Shelwien]

> Command line based, with five executables to choose from(!?).

Yes, with recompression and without.

> Although command syntax is listed when running any of the executables with
> no parameters, it is not entirely clear which executable I should use to
> create a patch based on differences between two files?

Just read the first line from each utility?

hdiffz  [options] oldPath newPath outDiffFile
hpatchz [options] oldPath diffFile outNewPath

> Frankly I find these executables difficult to understand and use!?
> Are there no "driving instructions" for these executables?

Just run hdiffz > hdiff.txt and read hdiff.txt

> Are there easier to understand and to use alternative file differential patchers rather than this one?

Not really. There're plenty of such utilities: https://en.wikipedia.org/wiki/Delta_encoding#VCDIFF
But most have some problems with multi-GB files etc.

> Preferably with GUI?

Maybe "Beyond Compare"? I never needed one, but you can look.
In any case, do you expect it to generate an sfx for you or what?

[brispuss]

Thanks. But the nomenclature/syntax is slightly confusing (for me).

No sfx required.

But going back to an earlier reply -

> Quickbms can only update existing compressed files

Its also not an issue, since you can zero out the place used by extracted files in the container.

What exactly do you mean by that? Can you explain in a bit more detail please?

[Shelwien]

quickbms only needs the container headers, both for export and reimport.
So if we'd zero out the compressed data in the container file, this "empty" container can be compressed to a small enough size,
and then you can unpack it and reimport recompressed files.
As to specifically making this empty container, I don't have an universal solution - quickbms prints offset/size for exported files,
so I'd make a script which would use these.
For example, this is something similar: http://nishi.dreamhosters.com/u/lzmadump_v1.rar

Afaik, there's an utility specifically for this, that takes a filelist, and removes the data of files in this list from
a given container file, but I don't have it.