precomp error: std::bad_alloc

[lohem]

Hi all,

I ran precomp on 4GB archive, and it gave error:

94.21% \terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc
This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information

Can anyone help me resolve error?

[schnaader]

Using verbose mode (precomp -v ...) could help to get more information on what leads to that crash and which stream causes it.

Looking at the error message, it looks like this is the Linux version, right? If so, do you have a Windows system available and can you try to run the Windows version of Precomp over that file?

You can also try to split the file into smaller blocks (50 or 100 MB in size) with some archiver and run Precomp over each block, so that if one of the block still gives the error, it would be easier to upload somewhere so I can have a closer look at it.

[Shelwien]

Its likely a failed new[] somewhere (out of memory).
Could be packjpg, as usual

[m^2]

I can only suggest to use Valgrind.
It's awesome for such bugs.

[schnaader]

Its likely a failed new[] somewhere (out of memory).
Could be packjpg, as usual

PackJPG isn't available in the Linux version, so I think it could be something different this time (if the bug really appears in the Linux version). Also had a similar bug directly after porting Precomp, it seems that Windows/MinGW doesn't care about fclose on an already closed file, but Linux/g++ does and throws a runtime error.

If it really is an out of memory problem, though, it would be harder to reproduce and there either would be a big memory leak or a wrong allocation where e.g. a signed int is misinterpreted. My best guess atm are the GIF routines where space for the image data is allocated, all other allocations are for initialization of fixed size buffers and filenames.

I can only suggest to use Valgrind.
It's awesome for such bugs.

Yes, I already fixed some minor memory leaks with it. The downside is that it further slows down Precomp so I doubt it would be helpful on a 4 GB test file, but perhaps it can be reduced on a smaller testfile.

[lohem]

Using verbose mode (precomp -v ...) could help to get more information on what leads to that crash and which stream causes it.

I will post back the results. (takes a lot of time even though i have added the -t-j option.)

Looking at the error message, it looks like this is the Linux version, right? If so, do you have a Windows system available and can you try to run the Windows version of Precomp over that file?

Im using Windows.

[lohem]

(94.21%) Possible GIF found at position 10629784
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc

will try with -t-fj

[schnaader]

(94.21%) Possible GIF found at position 10629784
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc

will try with -t-fj

Yes, this is what I suspected. There is some data that looks like a GIF header (which has a high chance of false positives, as the main marker is "GIF87a" or "GIF89a" and can appear e.g. in text files, too), but it seems there is no GIF. Some of the following bytes are interpreted as width/height of the image and this value gets too large, so new() tries to allocate a ridicolous large amount of memory (somewhere between 1 and 4 GB).

"-t-fj" should work in this case, another possibility would be "-i10629784" to ignore this particular stream.

It still would be nice if you could try to split the file into smaller parts (50-100 MB) and upload the crashing part somewhere so I could reproduce and fix the bug.

[lohem]

I am backing up my personal data, so i am not comfortable uploading part of it online. And, already a lot of days have been spent on this 4gb bundle. Yet, the job is not over.
apart from the GIF bug, precomp is hanging on zlib-streams. (two times).
It would be helpful if i could resume the process after adding the ignore position.
Although, i wont be breaking down the bundle, I can run your fix on the whole thing and report back the results.

Edit: It again hanged(at 67%) on a zlib-stream which it earlier was able process.

[schnaader]

I am backing up my personal data, so i am not comfortable uploading part of it online. And, already a lot of days have been spent on this 4gb bundle. Yet, the job is not over.
apart from the GIF bug, precomp is hanging on zlib-streams. (two times).

OK, in this case I'll try to create an artificial GIF test case myself. By the way, if you use Precomp for backups, you should make a checksum of the original file (MD5, SHA1...) and compare it with the restored file. There have been some rare edge cases in the past that led to incorrect recompression e.g. for PNG files. At the moment, Precomp doesn't store any checksums (but will do so in later versions). Also make sure that you keep any tools needed to restore the backup, especially the Precomp binaries as it isn't backwards compatible yet.

As for the problems with zLib streams, I can only guess. Perhaps there isn't enough free diskspace anymore, although Precomp should give a clear error message in this case. Also, did it only hang or did it crash? Some of the streams can take long to process. The activity indicator isn't visible in verbose mode and even if it is active, there are some actions that still block it so it won't move for a while.

Another thing comes to my mind: You should be using "-c-" to disable the built-in bZip2 compression and compress the PCF file with your favorite tool afterwards (e.g. 7-Zip). This will generate a PCF file larger than the original, but make the precompression faster and lead to a smaller compressed file.

[lohem]

you should make a checksum of the original file (MD5, SHA1...) and compare it with the restored file.

yes, i always do that.

especially the Precomp binaries as it isn't backwards compatible yet.

good that you mentioned. I assumed it was backwards compatible.

As for the problems with zLib streams, I can only guess. Perhaps there isn't enough free diskspace anymore,

There is more than enough disk space & RAM & CPU usage too.
Is there a way to increase or specify precomp's RAM & CPU usage?

although Precomp should give a clear error message in this case. Also, did it only hang or did it crash?

It gave mesg when the gif error occured. Other times(zlib) it just stops working. Then, I get windows mesg that precomp has stopped working.

Another thing comes to my mind: You should be using "-c-" to disable the built-in bZip2 compression

Yes, i do that.

The weird thing is why precomp stops-working on position that it was able to process on a previous run.

[schnaader]

It gave mesg when the gif error occured. Other times(zlib) it just stops working. Then, I get windows mesg that precomp has stopped working.

That's strange. There were similar errors (zlib streams crashing although they worked fine in previous runs) with Precomp 0.4 that didn't use ZLIB1.DLL, which is why it was reintroduced and has to be in the same path as the Precomp binary. Since you're using the newest version, I can only imagine that there's some undetected bug with files larger than 2 GB or with rare zLib streams that causes this behaviour.

[lohem]

yes, its strange. i did not have so many problems on one archive while using precomp before.
Since, i am still not getting the complete output, i would just 7zip-it for the time.
Hope bringing out this issue helps in making the next version better.