Error recovery from archive?

[roytam1]

Besides RAR, which archive supports error recovery?
Will you add such feature when you are designing archiving file format?

[m^2]

FreeArc, Sqx.

Don't know about Sqx, but both FA and RAR are not reliable here, tiny damage in unfortunate place destroys all protection. I find it very sad, because this is a really useful function.

[Shelwien]

There's http://en.wikipedia.org/wiki/Par2

As to rar, can you tell me where's that "unfortunate place"?
Once I tried to break it, but failed - it even handled some deleted and inserted bytes.

[roytam1]

There's http://en.wikipedia.org/wiki/Par2

As to rar, can you tell me where's that "unfortunate place"?
Once I tried to break it, but failed - it even handled some deleted and inserted bytes.

IMO it is hard to say PAR(1/2/3) as "file archives".

[m^2]

There's http://en.wikipedia.org/wiki/Par2

As to rar, can you tell me where's that "unfortunate place"?
Once I tried to break it, but failed - it even handled some deleted and inserted bytes.

I got logs somewhere, but now I don't I guess that they got lost when my pendrive broke this summer. I did a quick and rough test to only find a bug where recovery failed to notice and fix a broken recovery record, not data breakage.

[Shelwien]

Yes, but I guess they think that recovery is a method to fix an archive when you can't unpack it.
Restoring lost recovery records probably requires rebuilding them from data, which is slow, so they don't do it.

[m^2]

Yes, but I guess they think that recovery is a method to fix an archive when you can't unpack it.
Restoring lost recovery records probably requires rebuilding them from data, which is slow, so they don't do it.

Your recovery records might bit-rot and rar doesn't let you know even when you ask it to. This is very wrong even if it's faster.

[Matt Mahoney]

Is this really a useful feature? I mean, how often do you see files lose just a few bits of data?

[m^2]

I left my bruteforce script running and got another bug:

Building fixed.tst1.rar
Scanning...
Data recovery record found 100%
Sector 108 (offsets D800...DA00) damaged - cannot recover data
Reconstruct archive structure ? [Y]es, [N]o

Sounds bad and like said, cannot recover the original rar. Nevertheless data is intact.

ADDED:
Another kind of bug:

Building fixed.tst1.rar
Scanning...
RR- ??? - the file header is corrupt
RR- ??? - the file header is corrupt
Data recovery record not found
Reconstructing tst1.rar
Building rebuilt.tst1.rar
Found  avutil-51.dll
Done

Again, data is intact.

[Shelwien]

@Matt:
Yes, its a very useful feature.
In fact its a "must have" for large archives transferred via non-ECC channels (like http/ftp or usenet).
And of course its not a matter of "a few bits". Archive recovery is based on methods similar to what's used in RAID, and is able to recover from loss of a few sectors.
Anyway, I'd suggest to download rar and test something like
- rar a -rr1% archive data
- break the archive with a hexeditor (stuff like deleting 512 bytes is ok)
- rar r archive
- rar t archive

[m^2]

Bingo. A single byte corruption that breaks an archive.

Is this really a useful feature? I mean, how often do you see files lose just a few bits of data?

We posted at the same minute and I missed your question.
There are many use cases:
1. Optical backup. A little dated already, but discs loose readability over time, cheap disks often after few years. Initially internal error recovery algorithms work OK, but over time they become insufficient and storing some more of your own is helpful. And more importantly - it's free, your backup often don't take whole disks but only some portions of them and you can either leave the rest empty or fill it with something useful.
2. Consumer HDDs have bit error ratio of 10e-15 - 10e-14. That's a lot and I've seen corrupted files on my disk. ADDED: CERN made a study on silent data corruption some time ago and found that....firmware bugs in their controllers caused much higher silent error ratio then they should get. So such things happen too. ZFS FTW. ADDED: SSDs have different rotting patterns than HDDs, flash itself is much worse, but controllers actively manage it and move data from time to time. But consumer SSDs are often flaky and you can guess whether producers put a lot of effort for such invisible features.
3. Cyberlockers. I don't know why, but they are notoriously unreliable, if you use them a lot, you have to be prepared because sooner or later they will loose your data.

These are only the uses that I had personally, there are likely others too.

[Shelwien]

Thanks, this is interesting.
Apparently it fixed the actual broken byte, but then inserted a copy of 0xB8 bytes
around the broken byte (0xD748-0xD7FF) at the start of the next sector (0xD800).
This may be a bug, why don't you report it

[m^2]

This may be a bug, why don't you report it

Because I don't care. :P

[Surfer]

KuaiZip have recovery records, AFAIK it is LZMA + simple Reed-Solomon codes.

Also, interesting video about new tool
http://www.youtube.com/watch?v=Rf1c0BA-OXk

[roytam1]

KuaiZip have recovery records, AFAIK it is LZMA + simple Reed-Solomon codes.

Also, interesting video about new tool
http://www.youtube.com/watch?v=Rf1c0BA-OXk

Try comparing with MultiPar as QuickPar has no further development in these years.

[Surfer]

Try comparing with MultiPar as QuickPar has no further development in these years.

You can ask author on youtube or try it for yourself.