The best way to compress 2 mostly idenctical VMs?

[nimdamsk]

Hi all.

I need to compress 2 VMs to upload them to remote location.
VMs are Windows Server 2008 R2 with latest updates. 1st is Standard ed., 2nd - Enterprize. Files are from VMware Workstation.
The host to compress files will be 32-core 128GB RAM Xeon. Decompression will be done on 24 core 56GB RAM Xeon.
I've made archive this way: zpaq -m0 (to dedup) | srep -m3 | (nanozip -D) or (tor -16). Compression was done on old Pentium E2200 at night.
LzTurbo -39 and -49 and tor -7 gave bigger file, so i've skipped them. tor -16 used 11,6 times of nanozip to make 3,7% less archive.
Anyway, i've had already tor -16 file, it was smaller and decompression (i thought) had to be faster, so i used archive from tornado.
From 34210MB VMs zpaq made 20930MB file, then srep squeezed to 13325MB and tornado to 3735MB.
By the way: export from VMware Workstation to compressed ovf made 13570MB files (3,5 times bigger than zpaq+nanozip) and took slightly more time.
WinRAR made ~9GB archive.
So, here we go for why the topic was made.
On the reciever side (32-core 128GB RAM Xeon) decompression of tor and srep took resonable amount of time. But 64-bit zpaq took > 60 min to decompress files. On 32-core machine.
I've also tried zpaq alone with -m 4 option, it took 40 min, all 32 cores and 14GB RAM to create 4647MB file.
And the question is: is there a way to compress these VMs to <= 4GB file with pack\unpack speed >= 30 MB/s and not using temporary files?
I wanted to use shar | srep | any fast archiver capable to use 16+ cores, but shar | srep gives an error "input file is larger than file specified" (shar alone works OK). Tornado, nanozip, LzTurbo, 4x4, fazip - all of them either don't support both stdin\stdout for data or don't use all cores on pack\unpack.

Any help is appreciated, thanks.

[Matt Mahoney]

What about just zpaq -m 1? For -m 3 and higher, decompression speed is similar to compression (slower). -m 2 decompression should be as fast as -m 1 but compression is slower. -m 1 is sometimes faster than -m 0 because the compression is faster than disk I/O.

You can also try smaller fragment sizes to improve dedup like -fragment 2 instead of the default -fragment 6 (4 KB average size instead of 64 KB). But you have to use the same -fragment option for all updates for dedup to work. Smaller fragments may increase the archive size if there are not a lot of small fragments to dedup, and also increase memory usage. The default needs about 0.1% of the archive size in memory for the fragment tables.

I found that speed doesn't scale to very large number of cores like 24 or 32 for the faster methods. It might be faster with -threads 8 or even -threads 4. You can also try larger blocks like -method 16 or 28. The default for 1 is 14 (16 MB). For 2 and higher it is 26 (64 MB).

[Bulat Ziganshin]

not using temporary files

i think you can't do it with srep. btw "fazip 4x4:t32:tor" should use all cores and compress stdin to stdout

[Jarek]

The best way to compress 2 mostly identical VMs

To compress two "mostly identical" sources, in theory you need
H(X,Y) = H(X) + H(Y) - I(X;Y)
where H is Shannon entropy (sizes of separate archives) and I is mutal information - capacity you could safe thanks to their similarity.
So you could encode one of them, then encode the "difference between them", which in the simplest case (no synchronization errors) is just XOR of two sources (mostly zeros - can be well compressed).
However, there probably are synchronization errors in your case, so you need a smarter way for find and represent the difference between these two files: something like "delete block from a to b, insert block '...' in position c" etc.

In above case the encoder would need to have both files.
It turns out that theoretically we could get the same capacity if encoders of both files cannot communicate (no way to find "the difference") - it is so called distributed source coding problem, like Slepian-Wolf or Wyner-Ziv.
However, it becomes more complicated and costly ... and adding synchronization errors would make it a nightmare ...

[nimdamsk]

I'll try fazip and zpaq again, as suggested. And may be Bulat's fb too.
Theoretically the best way to compress these VMs will be parsing internal NTFS and making links to repeated files. Or making backup from inside of powered on VMs to dedup storage. Now their virtual disk images are not clean, there are unused, but not zeroed data, fragmented data, may be compressed\sparse files, may be pagefile. Sometimes there is hibernation file too. All these moments can be taken if specialized backup utility will be made. Now rolling dedup from zpaq can eleminate some of the problems, but not all. And it takes time.

[Bulat Ziganshin]

fazip by itself doesn't provide deduplication and fb ATM doesn't decompress at all so youк only choice is zpaq/exdupe. and you can use VMWare command to clean VM files prior to compression

[nburns]

To compress two "mostly identical" sources, in theory you need
H(X,Y) = H(X) + H(Y) - I(X;Y)
where H is Shannon entropy (sizes of separate archives) and I is mutal information - capacity you could safe thanks to their similarity.
So you could encode one of them, then encode the "difference between them", which in the simplest case (no synchronization errors) is just XOR of two sources (mostly zeros - can be well compressed).
However, there probably are synchronization errors in your case, so you need a smarter way for find and represent the difference between these two files: something like "delete block from a to b, insert block '...' in position c" etc.

In above case the encoder would need to have both files.
It turns out that theoretically we could get the same capacity if encoders of both files cannot communicate (no way to find "the difference") - it is so called distributed source coding problem, like Slepian-Wolf or Wyner-Ziv.
However, it becomes more complicated and costly ... and adding synchronization errors would make it a nightmare ...

bsdiff+bspatch is what I would use for the diff+patch.

[nimdamsk]

I don't really need diff now. When needed (i.e. receiver has Windows SP1, need to upload Windows SP2) i successfully used xdelta3.

[nimdamsk]

Hmm... Somehow zpaq -m 0 archive was unpacked in 14 min on 24 core 56GB RAM Xeon. Instead of >60 min on 32-core 128GB RAM Xeon.
May be NUMA issues, may be LUN throughput, may be I\O latency. I'll try to find the problem if have some time.
But current scheme (zpaq -m 0 | srep -m3 | tornado or nanozip or fazip) seems to be acceptable.

Thanks all again for excellent soft.

BTW, is it possible to make selfmade EMC DataDomain using opensource?

[nburns]

To compress two "mostly identical" sources, in theory you need
H(X,Y) = H(X) + H(Y) - I(X;Y)
where H is Shannon entropy (sizes of separate archives) and I is mutal information - capacity you could safe thanks to their similarity.
So you could encode one of them, then encode the "difference between them", which in the simplest case (no synchronization errors) is just XOR of two sources (mostly zeros - can be well compressed).
However, there probably are synchronization errors in your case, so you need a smarter way for find and represent the difference between these two files: something like "delete block from a to b, insert block '...' in position c" etc.

In above case the encoder would need to have both files.
It turns out that theoretically we could get the same capacity if encoders of both files cannot communicate (no way to find "the difference") - it is so called distributed source coding problem, like Slepian-Wolf or Wyner-Ziv.
However, it becomes more complicated and costly ... and adding synchronization errors would make it a nightmare ...

Giving this problem another look --
I don't think Slepian-Wolf is relevant. The OP apparently has both files on the machine used for compression, so he could just concatenate them and use any compressor. Some kind of LZ with unbounded match distances seems appropriate. From a theory standpoint, concatenating doesn't sacrifice the possibility of achieving optimal compression.

"However, there probably are synchronization errors in your case, so you need a smarter way for find and represent the difference between these two files: something like "delete block from a to b, insert block '...' in position c" etc."

If you assume the files are concatenated, long-range LZ provides a solution to that problem. Rather than a sequence of edits (insert, delete, substitute), it's a sequence of prior matches to copy from, which ultimately isn't a big difference.

[nburns]

I'll try fazip and zpaq again, as suggested. And may be Bulat's fb too.
Theoretically the best way to compress these VMs will be parsing internal NTFS and making links to repeated files. Or making backup from inside of powered on VMs to dedup storage. Now their virtual disk images are not clean, there are unused, but not zeroed data, fragmented data, may be compressed\sparse files, may be pagefile. Sometimes there is hibernation file too. All these moments can be taken if specialized backup utility will be made. Now rolling dedup from zpaq can eleminate some of the problems, but not all. And it takes time.

Cleaning the images is probably the lowest-hanging fruit for making smaller files. Making repeated files into links inside NTFS should not necessarily help much -- you'd just be doing the same thing as the compressor, but manually and using NTFS. However, zeroing out unallocated parts of the disk is absolutely a good idea, since it's 100% waste, yet it can take up large amounts of space in the compressed images. Defragmenting the filesystem is a good idea, because it's likely to help and it's automatic -- the machine does all the work. Pagefiles/swap space should be deleted and zeroed out. The way compression works, at a high level, is by finding segments that are repeated in multiple places and therefore take up x*n bytes, and compressing it down to something like x+n bytes, by keeping one copy and replacing the others with references. That means that you shouldn't worry too much about extraneous extra copies of important data; the biggest savings is from removing unimportant data completely -- especially audio, video, and already compressed data, which can't be compressed further.

I don't really need diff now. When needed (i.e. receiver has Windows SP1, need to upload Windows SP2) i successfully used xdelta3.

When you have two files that are almost identical, you can replace one of them with a diff. Concatenating them and using a long-range compressor should have a similar effect, but bsdiff is highly optimized and might do better.

Hmm... Somehow zpaq -m 0 archive was unpacked in 14 min on 24 core 56GB RAM Xeon. Instead of >60 min on 32-core 128GB RAM Xeon.
May be NUMA issues, may be LUN throughput, may be I\O latency. I'll try to find the problem if have some time.
But current scheme (zpaq -m 0 | srep -m3 | tornado or nanozip or fazip) seems to be acceptable.

I'd start by looking at the cpu architectures. It's hard to tell from the info you included. All I see is that both machines have more than enough cores and that memory isn't the problem.

[Bulat Ziganshin]

> zpaq -m 0 | srep -m3 |

why zpaq? if yoг allow temfpfiles, just tar data together and compress them with srep. i.e. "tar ... | srep -m3 | ..."

[a902cd23]

You can try EXDUPE (www.exdupe.com).

Use with -stdout to apply compression from other program if you like.

exdupe -g16 -x0 -t1 *.WMs -stdout | rar a -m5 -md1g -ma5 -si FILES.RAR
rar p -inul FILES.RAR | exdupe -R -stdin EXTRACTED\

[nimdamsk]

> zpaq -m 0 | srep -m3 |

why zpaq? if yoг allow temfpfiles, just tar data together and compress them with srep. i.e. "tar ... | srep -m3 | ..."

You are right. Srep made 2% smaller file and seems took less time, but not sure here, had no time to look carefully.
May be i'll have some more time for tests. Unfortunately those systems are not ours and have some productive software running.
But it's interesting to test all suggestions here.

[Matt Mahoney]

zpaq computes SHA1 hashes for dedup and data integrity and checks them on extraction. Otherwise with -m 0 it is just copying data. You can try -fragile to turn off the integrity checks (but not dedup).

[nimdamsk]

zpaq computes SHA1 hashes for dedup and data integrity and checks them on extraction. Otherwise with -m 0 it is just copying data. You can try -fragile to turn off the integrity checks (but not dedup).

Thank you Matt, i'll try. BTW i've made mistake: 1-st machine (32 core 128GB) is not Xeon, but AMD Opteron. This may explain why is it may be slower than 24 core Xeon on 2-n machine. I'll rerun tests on 24 core Xeon some days later.

[nburns]

Thank you Matt, i'll try. BTW i've made mistake: 1-st machine (32 core 128GB) is not Xeon, but AMD Opteron. This may explain why is it may be slower than 24 core Xeon on 2-n machine. I'll rerun tests on 24 core Xeon some days later.

Unfortunately, AMD got left behind in terms of technology. That's the likely culprit, especially if the AMD CPU is older.

[fcorbelli]

If you are very, very brave, you can take a rather naif approach.

1) use snapshot.exe to take the images of the VMs (all partitions)
2) take only the first one (the "boot")
3) extract from 1) file-by-file in two directory, then deduplicate&pack

To restore, first remake partition table & boot (2), then the system disk (empty).
Then extract file-by-file (from 3).

Not so straight, but I think this should be the most efficient (in space term), if you don't care for ACLs.

Only a gekandenexperiment...

Or you can mount a iSCSI-zfs volume (from a solaris box, for example) on the VM disk, then try zfs's send