Precomp 0.3.8

[osmanturan]

@Intrinsic:
Most of my friends use TIFF for exchanging printable materials (like brochure, poster etc) over the internet (typical poster around 40-60 MB). Some designer programs don't support deflate mode. So, they use LZW or raw mode. Note that, all of these large TIFF files in CMYK mode.

[Intrinsic]

Yeah maybe it's just best to leave it in slow mode then? i knew it(Deflate) wasn't used that widely(i myself just leave them raw), depends on how easy it is to add i guess

[maadjordan]

tiff has two deflates one is zlib other is adobe and you can refer to adobe tiff- spec for description.. lzw,lzw+predictor,jpeg,packet compression support is not supported yet even with v0.3.8.. i can wait for its support

[Intrinsic]

Well, the original TIFF was done via Hugin, i then sent the image to a friend who saved it out under CS3 using TIFF with ZIP chosen:

Original filesize: 1,440,868 bytes
Precomp + CCM: 1,187,175 bytes

Recompressed streams: 5/5
JPG streams: 1/1
zLib streams (slow mode): 4/4


I did the same with The GIMP:

Original filesize: 1431456 bytes
Precomp + CCM: 1,180,735 bytes

Recompressed streams: 13/13
zLib streams (slow mode): 13/13

Oddly enough, when i tried the same with IrfanView precomp couldn't find any streams to recompress hmm.

[schnaader]

Include decompression for LZMA streams.
Actually a lot of Linux distribution include LZMA compressed isos.

I'll have a look at it, but at the moment new algorithms get to the end of my to-do list, so don't expect it for the next version

if lzma support is on the way, please consdier .RH 3d format..
i noticed that right hemisphere format .rh from http://www.righthemisphere.com support LZMA compression here is samples.. http://rapidshare.com/files/13413608...Models.7z.html (link temporary for 30 days..)

I'm not sure about LZMA in there - do you have any specifications or something that would approve this? At least both of the samples you provided can be precompressed using -slow.

Is it possible to add TIFF files to the list of ones that precomp will check without using -slow(or maybe it'll have to anyways?)? TIFF files can use Deflate and when precomp -slow is used on these types it makes a huge difference as you'd expect.

This actually has a chance to get into the next version . Anyway, some sample files where Deflate is used would be nice, indeed.
As maadjordan said, TIFF offers some other compression methods, a quick glance over the specs reveals at least PackBits (some RLE stuff), modified Huffman and LZW. These should be more widely used, but are harder to implement, so same as for LZMA here - patience

[Intrinsic]

At this link below i've uploaded a file saved out with various programs: Original VIA Hugin, The GIMP, PS CS3, IrfanView.

http://www.zenadsl5706.zen.co.uk/TIFF%20Deflate.rar

Can provide more examples if needed just say. And i ran precomp over the IrfanView saved file again, and this time it picked up the zLib streams! No clue why it didn't wanna do that yesterday although it was on a different XP setup. Seems a bit overkill though on the streams and maybe a bug in IrfanView? I'll have to bung Irfan an email as he's fixed some other bugs i've found recently

Recompressed streams: 628/827
zLib streams (slow mode): 628/827

Heh.

[schnaader]

At this link below i've uploaded a file saved out with various programs: Original VIA Hugin, The GIMP, PS CS3, IrfanView.

http://www.zenadsl5706.zen.co.uk/TIFF%20Deflate.rar

Thanks, that'll be enough to start TIFF support without slow mode.

Recompressed streams: 628/827
zLib streams (slow mode): 628/827

Heh.

Same here... It seems like IrfanView just compresses the image data in small chunks (around 3000 bytes) instead using some big zLib chunks (for example Hugin uses chunk size around 1 MB).

Edit: I've found out why this happens. There are some very small zLib streams at the end of the IrfanView TIF file (each around 20 byte) and the new Precomp only recompresses streams bigger than 64 byte in slow mode to avoid false detection. Using Precomp 0.3.7, all streams are recompressed. However, these are just many black pixels, so it doesn't help to improve compression ratio, the UHArc archive even grows by 100 bytes.

Some other interesting things:
Hugin and IV generate relatively big files (~2.1 MB) while GIMP and Photoshop files are around 1.4 MB. This could be either some additional data (alpha channel or such) or just better zLib compression.

The image data size is around 3.5 MB, PCF files for PhotoShop and GIMP are of that size, Hugin is pretty big with 4.6 MB and IrfanView seems to be smaller because of the many unrecompressable streams (2.6 MB) (Edit: Using Precomp 0.3.7 this also decompresses to ~3.5 MB).

Compressing the PCF files with CCM, Hugin has the worst size (1.31 MB), even IrfanView (1.29 MB) is better, PhotoShop and GIMP are both around 1.18 MB... Anyway, this seems to be CCM-related, because everything chages when using UHArc: Hugin is 1.19 MB, GIMP and Photoshop 1.18 MB, IrfanView gets down to 1.16 MB (!)

[Intrinsic]

Yeah i noticed the size difference, i guessed that it was down to different zLib compression levels. One thing that the Hugn image does contain when loaded into The GIMP is that the black is actually transparent, but that wouldn't account for the bigger file size so it must just be a faster/lower zLib compression mode/setting.

[SvenBent]

What about RLE support in .PCX/.BMP
its a pretty simple algo.
But maybe decompressiong before compression is not going to give any gains?

I'll will try test BMP in raw vs RLE encoding tonight.

should i test Winrar and 7-zip or CCM and RZM ?

[SvenBent]

Just a quick test.
i wanted to try .BMP/RLE vs .BMP/RGB, however BMP/RLE seem to only work with 256 bit colors. and my test pictures was the truecolor kodak test suit.

So instead i tried .PCX vs. BMP/RAW

Again this is the 24 truecolor reallife pictures from Kodak Lossless True Color Image Suite


BMP 24 files = 28.312.848 bytes
7-zip ultra = 14.157.073 bytes
Winrar Best = 13.310.350 bytes
NanoZip -cO = 13.311.653 bytes

PCX 24 files = 28.128.988
7-zip ultra = 18.889.078 bytes
Winrar Best = 20.935.361 bytes
NanoZip = 17.378.627 bytes

Even though the files gain only a little by the RLE compressions (RLE is bad on reallife pictures) its seem to really hurt further compression.

I will try to find some 256 colors "drawn" pictures to test next.

[Intrinsic]

Here is a bunch of hand drawn 256 colour graphics for you: http://www.zenadsl5706.zen.co.uk/256...ixel%20Gfx.rar

Oh...just noticed the angel particle picture isn't 256 colours in that archive, so you can exclude that. Has some nudity(Just breasts) in them hope you don't mind, if you do i can provide plenty of other examples.

[SvenBent]

just another quick test with the pictures form Intrinsic (The angels pixtures was reduced 256 colros with dithering)

this time it used BMP/RLE vs BMP/RGB


BMP/RGB 7 pictures = 2.068.986 bytes
7-zip ultra = 705.113 bytes
WinRAR best = 780.812 bytes
NanoZip -cO = 594.936 bytes

BMP/RLE 7 pictures = 1.689.696 bytes
7-zip ultra = 819.095 bytes
WinRAR Best = 887.394 bytes
NanoZip -cO = 728.619 bytes


Once again it seems to really help on compression by removing the RLE encoding first

--- edit ---
Targa (.TGA) files can also containe RLE encoding, and are used in a couple of games

[schnaader]

Especially in the truecolor example there's much to save, yes. I always delayed RLE recompression because I didn't know if the ratio gain would be worth it. Worst cases for RLE (truecolor) can gain much compression, but not always as much as storing them as non-RLE and compressing them.
When recompressing, Precomp needs to store the RLE lengths, too, if they're not optimal. So there is no problem if 1024 black pixels are stored as (1024,0), but they can also be stored (to gain speed or random access or just because the encoder is crap) as something "evil" (=unexpected) like (2,0),(3,0),(4,0),...,(43,0),(44,0),(35,0).
For optimal coded images or some other special cases that can be prepared (i.e. no RLE at all even if it would be possible), everything is fine and we just have the image data and some little overhead, but in other cases we could have to store all the lengths, too. This would create almost the same file as the RLE-coded image, except that the image data and the RLE lengths get splitted, but this won't help compression that much.
Anyway, I can't imagine any worst case scenario where recompression would hurt the ratio, so it's worth a try as it is relatively simple to implement.

[SvenBent]

Yes i kinda thougth about the non optimal RLE values being a problem.
And my "experiments doesnt take that into acount, because i actually throw the RLE away with no regards for keeping it later.



For the files without optimal RLE encoding we could mabye detect if there seem to be a max lengt of the RLE encoding. We could detect if the RLE "stream is" that is logner then x is actually encoding at x + new stream

E.g 300 black pixels are 255,0 45,0 if there is alot of 255 + new value we migh just save the RLE_maxlengt = 255 and then throw every info about RLE encoding with value off 255 aways as well.


Please note that I have not compression programming skill.
but i could think the non optimla RLE encoder would use some small buffer to get speed gains and therefore have the same artificial max of RLE lengt throghout the entire file.

if that is the thruth.
Then most nonoptimla RLE encoded pictures would only need the maxlengt information and we could throw out all RLE information.


-- edit --

if its not true regarding the maxlengt of RLE encoded "streams"
i might have another idea. but since I'm at work a better do what I'm paid for... will be back with #2 idea soon.
just a quick tip: "RLE encoding the RLE information"

[SvenBent]

Hmm some more testing reveals that removing RLE encoding hurts compression in 7-zip and Winrar.

These 256c pictures contains some primitive dithering in gradient areas. but it also contains large block of the same colors which are perfect for RLE.. it was kinda these type of pictures i believe would be typical for RLE encoding pictures.
its taking for some H game named Sentimental Shooting

Anyway i tested TGA/RLE BMP/RLE BMP/RGB

BMP_RGB.7z   10,5 MB (11.084.364 bytes)
BMP_RGB.rar  10,1 MB (10.595.547 bytes)

BMP_RLE.7z   10,3 MB  (10.838.794 bytes)
BMP_RLE.rar   9,67 MB (10.146.599 bytes)

TGA_RLE.7z   10,4 MB  (10.934.653 bytes)
TGA_RLE.rar   9,66 MB (10.139.548 bytes)

RLE -> compression is better then RGB -> compression

[maadjordan]

another format to improve support is SKP (google sketchup) files as it contains raster+vector data .. the raster is pure "PNG" in which precomp does not recognize.

i have attached few samples to check..

i tested "fan.skp" with no luck in normal, -slow -brute mode .. all fail to identify the embedded PNG files with "bitmap ripper v1.08" at http://mark0.net/soft-bitmaprip-e.html i could extract these easily its 6.0+7.0+13.0 KB of 29 kb file so it takes most of that file.

precomp says there is three "png" streams but does nothing..
i hope you can improve "png" detection more.. thanks anyway

[maadjordan]

another format that precomp doesnot support is "window media player skins" *.wmz

samples are available at every windows version just look under

C:\Program Files\Windows Media Player\Skins

it normal zip file but not recognizable by precomp..

advzip can handle it with "advzip -z0 *.wmz"

[schnaader]

another format to improve support is SKP (google sketchup) files as it contains raster+vector data .. the raster is pure "PNG" in which precomp does not recognize.

precomp says there is three "png" streams but does nothing..
i hope you can improve "png" detection more.. thanks anyway

another format that precomp doesnot support is "window media player skins" *.wmz

samples are available at every windows version just look under

C:\Program Files\Windows Media Player\Skins

it normal zip file but not recognizable by precomp..

Yes, PNG/ZIP/zLib recompression has to be improved. Note that decompression always works correctly, it is just bit-to-bit-identity that can't be restored. This will hopefully go away if I parse zLib streams myself, but this will take some time and will definitely not change for the next two versions, sorry...

[maadjordan]

Yes, PNG/ZIP/zLib recompression has to be improved. Note that decompression always works correctly, it is just bit-to-bit-identity that can't be restored. This will hopefully go away if I parse zLib streams myself, but this will take some time and will definitely not change for the next two versions, sorry...

if lossy mode is introduce (as an option) it would great to me.. and i can wait for even three versions.. but please keep it in your todo list and be patient with me as i shall proceed with my precomp testing with other formats..

[nanoflooder]

Love Precomp, an awesome utility... But always wondered, is there any chance to see an option in the future of processing .cab (.msi) files? Would be just great!

[schnaader]

Love Precomp, an awesome utility... But always wondered, is there any chance to see an option in the future of processing .cab (.msi) files? Would be just great!

Yes, I also thought about this. I just saw there is a CAB SDK including some libraries available from Microsoft. Perhaps this will just work on CAB files, I'll have a look.

[maadjordan]

Yes, I also thought about this. I just saw there is a CAB SDK including some libraries available from Microsoft. Perhaps this will just work on CAB files, I'll have a look.

i suggest this one http://www.cabextract.org.uk/libmspack/ as it support more than cab(lzx..) please take it into consideration..

for other links use

http://www.speakeasy.org/~russotto/chm/
http://mateusz.free.fr/mscab/

[nanoflooder]

There's also some non-MS formats of CAB, like InstallShield 5, 6, 7... Is it possible for Precomp to be able to extract them as well?

[maadjordan]

first installshield CAB is different format than MS CAB.
second till today .. no way to expand or edit then restore the original data as the compression method still need more investigation..

the only version i could restore is InstallShield v3.0 which has formats .z,.1,.lib and its an old version .. all with icomp.exe or i3comp.exe or i95comp.exe

but look here for source isdec,i5comp,i6comp

ftp://ftp.uni-hannover.de/pub/mirror/os2/win32os2/

[schnaader]

first installshield CAB is different format than MS CAB.
second till today .. no way to expand or edit then restore the original data as the compression method still need more investigation..

the only version i could restore is InstallShield v3.0 which has formats .z,.1,.lib and its an old version .. all with icomp.exe or i3comp.exe or i95comp.exe

I'm not quite sure about this. I have experimented with various InstallShield/installation/CAB files and it seems that some of the files are not valid CABs, but only the header or some filename information is missing and is stored somewhere else and with some trial-and-error you can often decompress most of the data (or does this only work for older version?) Also, I only tried to decompress with WinRAR, perhaps this detects various formats which aren't the same.

Anyway, I guess one thing is sure: CAB won't be the easiest to implement... and the above is just for decompression, recompressing it bit-to-bit-identical will be a mess....

[nanoflooder]

Anyway, I guess one thing is sure: CAB won't be the easiest to implement... and the above is just for decompression, recompressing it bit-to-bit-identical will be a mess....

Yep, that's 100% true... I haven't seen any app yet to do that Apart from maybe some $999.99 stuff like "Setup Squeezer", which apparently usually don't work at all because of their super-duper technology of compressing all files into one 7z SFX

[maadjordan]

i have to withdraw my previous comment.. installshield is already supported with precomp in "brute" mode so on large files in needs to be improved.. is seems the compression is based on deflate in some how .. i'v attached an example of v5.0

[schnaader]

i have to withdraw my previous comment.. installshield is already supported with precomp in "brute" mode so on large files in needs to be improved.. is seems the compression is based on deflate in some how .. i'v attached an example of v5.0

That's very interesting, especially as WinRAR fails to extract that CAB. I downloaded i5comp and it asks for _sys1.hdr. If there is such a file, could you upload it, too?

[maadjordan]

here is a full package

ftp://ftp.net.pulawy.pl/pub/drivers/...D/APPLICATION/

here is a topic from "i6comp.txt" inside "i6comp020.zip"

1) No more ZDxxx.DLL - compression/decompression (deflate) routines are
statically linked in.

2)..

3) All IShield engine files are now stuck into data1.cab, so convertion to
single-volume cabinet has been changed. Conversion leaves data1.cab intact
and gathers everything else into data2.cab.
Data1.cab contains all installation and uninstallation files and
is used by Uninstaller, and data2.cab contains all 'user' files.

and take alook at zdata.lib i found "deflate Copyright 1995 Jean-loup Gailly"

.. it confirms thats its zlib and data1.cab should be same of same version for all ..

for data1.cab and data2.cab it works but in -brute mode only.. i tested http://www.traction-software.co.uk/d...chPrintPro.zip

[nanoflooder]

Interesting thing about the InstallShield CABs - Precomp does indeed unpack them in -brute mode, and that's amazing, considering it cannot unpack the standard Microsoft CAB files, I have just checked this... So maybe it's just the MS CAB recompression technology that needs to be implemented inside Precomp?..