parallel compression with batches

[evg]

Hello all,

after a bit of experimenting with pipes i found a fairly quick method of compressing huge files at the cost of lots of memory. The attached batches simply run both commands in parallel, usefull if you got memory to burn and at least a dual core machine. It also helps to overcome the 2GB process memory limit in XP32. At the bottom there are some measurings.

Attached you will find 4 batches for use with rep 1.2a and 7z 9.07/ccm 1.30c.
The batches are really crude, no error checking whatsoever, maybe somebody who is better at batches can elaborate.

Usage is simple: batch file_to_compress.

Unfortunatly there are not many compressors/filters which are able to use pipes for processing.

Maybe there is a wrapper to let normal compressors use stdin/out? I seem to dimly remember there was something similar for DOS/djgpp in the olden days.

Hope you find it usefull and best regards
evg

Source
Ubuntu karmic-desktop-i386.iso.pcf40, size 1,79GB (.iso 657MB)
precomp40 -slow -l3, took some hours

fa 0.52 Sept 8 2009
comp 419984KB, 2097 seconds, used memory ~1.4GB
fa decomp 177 seconds
-mrep:a99:b1gb:h24+exe+delta+lzma:128mb:max:a1:273: mc256

rep7z rep 1.2a+7za 9.07b
comp 419760KB, 1176 seconds, used memory ~2.5GB (Taskmanager)
unrep7z decomp 133 seconds
similar settings to fa, see batch

All files md5 checked out ok
Timings on Athlon X2 5600/4GB and XP32-SP3

[Bulat Ziganshin]

LOL just add -lc- to fa cmdline

[evg]

LOL just add -lc- to fa cmdline

Hello Bulat,

I tried out your suggestion but arc tries to allocate ~2.4GB memory and quietly fails without error message. This is quiet unsurprising because on a stock 32bit XP process memory is limited to 2GB minus several mb. I dont think that arc spawns a second process to overcome that limit. Maybe its possible with /3GB or /UserVA boot.ini switches but on my system this does not work out due to crappy drivers.

Also your suggested switch -lc- is neither documented in the program help or the online 0.40 translation. Maybe it would be time to update that too. I'd volunteer to help with the translation but my russian is nonexistent.

Dont get me wrong, I dont want to bash freearc, I've been using it intensively in the last time and find a very good archiver.

Best regards
evg

update:
rechecked my batch with 1536mb dictionary for rep, works quite fine too but reaches the limits of my system.
rep 1536m+7z 128m allocation in taskmanager: rep 1636724K (1598,36G) + 7z 1427512K (1394,05G) = total 3064236K (2992,42G)

[Bulat Ziganshin]

you are right, it's better to use 64-bit windows if you ahve 4gb of ram

[SvenBent]

For me personally i think these batches have a moot point and only have a benefit for people in af very specific HW layout


a: 7-zip LZMA2 can work with several threads anyhow.

b: this only utilize one extra core above 7-zip

b: they are for people that have to much memory and yet not enough.
They must have enough memory for rep + 7-zip yet not enough to just use the same dictionay with 7.zip



basically hey have a sweet spot with triple cores machine with a couple of gigs ram to spare (~4-6gb)


people with more ram just use 7-zip dictionary and not rep
people with more cores can just use more threads with 7-zip

my own batch for brute force comrpession uses 5-6 cores
the decoding batch uses around one core for each file to decompress



but basically Rep is getting useless for me, because for less then 1GB dictionaries i just use 7-zip
and Rep max dictionare is still "just" 2GB
so its only a few files rep show anybenefits to me