7GB ISO file compression (from DVD)

[anthalamus]

Hi,

I have found the following thread: http://encode.su/threads/1250-ISO-im...?highlight=iso but i don't think it's exactly what I need...

I have these big ISO files coming from copying DVD (actually nothing illegal, I ACTUALLY just want to back them up . I've tried to compress them using simple stuff like zip and bzip2 but the compression was not worth it (10% smaller only). I tried paq8l and zpaq but somehow it fails on them (says "Nothing to compress"/cannot find the file).

Anyone has a suggestion? time matters less to me (anything less than 24h is fine) but size does. what's the best i can expect?

Thanks!
Anthony

[sascha]

EDIT: Just recognized that not much of your post is pointing to movie dvds, which i handle here. If your DVDs are other stuff, than you need other ideas. But in general: if you want to gain more compression, we would need to know the data which is within this iso file. The methods in the other thread you mentioned are working really good for redundant data, as programs (e.g. Operating System) and maybe some games. But keep in mind that precompressed data is always hard to handle.

..........

Assumption: Movie-DVD material

..........

1. Because of the nature of the underlying codecs, which are already compressed, it's really hard for any general-purpose compressor to achieve good compression ratios.

2. You have to think about what you want: do you want bit-identical compression (iso is after compression/decompression 100% the same) or do you want visually-identical compression (bit's aren't the same, but all the "decoded" data is)?

If the latter is possible, than you can try some more domain-specific approaches; some non-structured remarks:

- Because the data is already compressed and all the general-purpose compressors have problems with this, it is a good idea to demultiplex the streams into elementary ones (video, audio1, audio2) and then proceed with these. In the decompression step you would need to multiplex them togeher and rebuild your iso (in general not bit-identical!).

- Nearly all movie dvds have a lot of fill-bits, just for beeing compliant with some hardware devices. Of course no general-purpose compressor will detect and remove them. Remultiplexing with a Matroska Container and the really good MKVToolnix program will ignore all these fill-bits resulting in a compression gain.

- DTS files can be compressed quite good, much more than AC-3 files. (just a random note for everyone who is bored: lossy DTS -> lossless audio => more compression gain than PAQ ; disclaimer: i'm not entirely sure if thats not a result of bad DTS decoders!)

- There is a research compressor in this forum (look for ocarina) which can give you up to 30% compression gains for MPEG2-Video. But yeah, it's only for testing at the moment, but demonstrating some kind of domain-specific recompression techniques. Read the ocarina thread for more info. There is no other project like that for MPEG2, AC-3, DTS (the main streams on movie dvds) that i know of!

So there are a lot of possibilities if you want to go the non-bit-identical way. If you want to do it that way, you will have to learn about all this multiplexing stuff and need to use some new software i suppose.

Some small example (it's only ONE example):

Movie ISO: 4.35 GB
Movie ISO 7-zip: 3.70 GB
Matroska Remux of Original ISO (demultiplex + multiplex): 3.89 GB
Matroska Remux + 7-Zip: 3.35 GB

maybe one of the best ways, but for testing/research only:
1. extract iso to video_ts folder
2. demux folder to elementary streams (one would have to check if the fill-bits are killed with demuxing!)
3. compress video stream with ocarina codec (thats the critical part; please don't use testing-marked codecs for archiving your movies)
4. compress dts-audio, ac3-audio, subtitles with something like 7-Zip or some fast symmetric compressor
5. tar all the files together, with all the needed information for rebuilding the video_ts folder and the iso file (e.g. audio-delays)

comment: one could see immediatly that this domain-specific approach does need a lot of steps/tools and there is a lot of stuff that can go wrong

maybe the easiest way:
1. use a commercial DVD->MKV remuxer (google for that if you want to find one)
2. 7-Zip (if you want to)
(still not bit-identical + you need a MKV->DVD remuxer)

[anthalamus]

thanks a lot for such a thorough answer!

i did not realize it would get complicated like that, I suppose the nature of the data does matter a lot. i will have to look more into what your suggesting, see if i can find a good compromise between result and difficulty. My dvd essentially contain lectures of about 30 minutes each, so I suppose it's comparable to a movie.
let me get back to you soon on what works best for me!

[Matt Mahoney]

Yeah, if you are compressing MPEG then most compressors will not do well because it is already compressed. paq8l will not handle files over 2 GB. Not that you would want to, given its speed. zpaq should handle large files, or else give you a warning (only in 32 bit linux). Not sure why it can't find the file.