Page 2 of 4 FirstFirst 1234 LastLast
Results 31 to 60 of 117

Thread: Precomp 0.3.8

  1. #31
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    @Intrinsic:
    Most of my friends use TIFF for exchanging printable materials (like brochure, poster etc) over the internet (typical poster around 40-60 MB). Some designer programs don't support deflate mode. So, they use LZW or raw mode. Note that, all of these large TIFF files in CMYK mode.

  2. #32
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    Yeah maybe it's just best to leave it in slow mode then? i knew it(Deflate) wasn't used that widely(i myself just leave them raw), depends on how easy it is to add i guess

  3. #33
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    357
    Thanks
    37
    Thanked 39 Times in 24 Posts
    tiff has two deflates one is zlib other is adobe and you can refer to adobe tiff- spec for description.. lzw,lzw+predictor,jpeg,packet compression support is not supported yet even with v0.3.8.. i can wait for its support

  4. #34
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    Well, the original TIFF was done via Hugin, i then sent the image to a friend who saved it out under CS3 using TIFF with ZIP chosen:

    Original filesize: 1,440,868 bytes
    Precomp + CCM: 1,187,175 bytes

    Recompressed streams: 5/5
    JPG streams: 1/1
    zLib streams (slow mode): 4/4


    I did the same with The GIMP:

    Original filesize: 1431456 bytes
    Precomp + CCM: 1,180,735 bytes

    Recompressed streams: 13/13
    zLib streams (slow mode): 13/13

    Oddly enough, when i tried the same with IrfanView precomp couldn't find any streams to recompress hmm.

  5. #35
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    631
    Thanks
    288
    Thanked 252 Times in 128 Posts
    Quote Originally Posted by lunaris View Post
    Include decompression for LZMA streams.
    Actually a lot of Linux distribution include LZMA compressed isos.
    I'll have a look at it, but at the moment new algorithms get to the end of my to-do list, so don't expect it for the next version

    Quote Originally Posted by maadjordan View Post
    if lzma support is on the way, please consdier .RH 3d format..
    i noticed that right hemisphere format .rh from http://www.righthemisphere.com support LZMA compression here is samples.. http://rapidshare.com/files/13413608...Models.7z.html (link temporary for 30 days..)
    I'm not sure about LZMA in there - do you have any specifications or something that would approve this? At least both of the samples you provided can be precompressed using -slow.

    Quote Originally Posted by Intrinsic View Post
    Is it possible to add TIFF files to the list of ones that precomp will check without using -slow(or maybe it'll have to anyways?)? TIFF files can use Deflate and when precomp -slow is used on these types it makes a huge difference as you'd expect.
    This actually has a chance to get into the next version . Anyway, some sample files where Deflate is used would be nice, indeed.
    As maadjordan said, TIFF offers some other compression methods, a quick glance over the specs reveals at least PackBits (some RLE stuff), modified Huffman and LZW. These should be more widely used, but are harder to implement, so same as for LZMA here - patience
    http://schnaader.info
    Damn kids. They're all alike.

  6. #36
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    At this link below i've uploaded a file saved out with various programs: Original VIA Hugin, The GIMP, PS CS3, IrfanView.

    http://www.zenadsl5706.zen.co.uk/TIFF%20Deflate.rar

    Can provide more examples if needed just say. And i ran precomp over the IrfanView saved file again, and this time it picked up the zLib streams! No clue why it didn't wanna do that yesterday although it was on a different XP setup. Seems a bit overkill though on the streams and maybe a bug in IrfanView? I'll have to bung Irfan an email as he's fixed some other bugs i've found recently

    Recompressed streams: 628/827
    zLib streams (slow mode): 628/827

    Heh.

  7. #37
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    631
    Thanks
    288
    Thanked 252 Times in 128 Posts
    Quote Originally Posted by Intrinsic View Post
    At this link below i've uploaded a file saved out with various programs: Original VIA Hugin, The GIMP, PS CS3, IrfanView.

    http://www.zenadsl5706.zen.co.uk/TIFF%20Deflate.rar
    Thanks, that'll be enough to start TIFF support without slow mode.

    Quote Originally Posted by Intrinsic View Post
    Recompressed streams: 628/827
    zLib streams (slow mode): 628/827

    Heh.
    Same here... It seems like IrfanView just compresses the image data in small chunks (around 3000 bytes) instead using some big zLib chunks (for example Hugin uses chunk size around 1 MB).

    Edit: I've found out why this happens. There are some very small zLib streams at the end of the IrfanView TIF file (each around 20 byte) and the new Precomp only recompresses streams bigger than 64 byte in slow mode to avoid false detection. Using Precomp 0.3.7, all streams are recompressed. However, these are just many black pixels, so it doesn't help to improve compression ratio, the UHArc archive even grows by 100 bytes.

    Some other interesting things:
    Hugin and IV generate relatively big files (~2.1 MB) while GIMP and Photoshop files are around 1.4 MB. This could be either some additional data (alpha channel or such) or just better zLib compression.

    The image data size is around 3.5 MB, PCF files for PhotoShop and GIMP are of that size, Hugin is pretty big with 4.6 MB and IrfanView seems to be smaller because of the many unrecompressable streams (2.6 MB) (Edit: Using Precomp 0.3.7 this also decompresses to ~3.5 MB).

    Compressing the PCF files with CCM, Hugin has the worst size (1.31 MB), even IrfanView (1.29 MB) is better, PhotoShop and GIMP are both around 1.18 MB... Anyway, this seems to be CCM-related, because everything chages when using UHArc: Hugin is 1.19 MB, GIMP and Photoshop 1.18 MB, IrfanView gets down to 1.16 MB (!)
    Last edited by schnaader; 7th August 2008 at 15:06.
    http://schnaader.info
    Damn kids. They're all alike.

  8. #38
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    Yeah i noticed the size difference, i guessed that it was down to different zLib compression levels. One thing that the Hugn image does contain when loaded into The GIMP is that the black is actually transparent, but that wouldn't account for the bigger file size so it must just be a faster/lower zLib compression mode/setting.

  9. #39
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    926
    Thanks
    58
    Thanked 116 Times in 93 Posts

    RLE support?

    What about RLE support in .PCX/.BMP
    its a pretty simple algo.
    But maybe decompressiong before compression is not going to give any gains?

    I'll will try test BMP in raw vs RLE encoding tonight.

    should i test Winrar and 7-zip or CCM and RZM ?

  10. #40
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    926
    Thanks
    58
    Thanked 116 Times in 93 Posts
    Just a quick test.
    i wanted to try .BMP/RLE vs .BMP/RGB, however BMP/RLE seem to only work with 256 bit colors. and my test pictures was the truecolor kodak test suit.

    So instead i tried .PCX vs. BMP/RAW

    Again this is the 24 truecolor reallife pictures from Kodak Lossless True Color Image Suite


    BMP 24 files = 28.312.848 bytes
    7-zip ultra = 14.157.073 bytes
    Winrar Best = 13.310.350 bytes
    NanoZip -cO = 13.311.653 bytes

    PCX 24 files = 28.128.988
    7-zip ultra = 18.889.078 bytes
    Winrar Best = 20.935.361 bytes
    NanoZip = 17.378.627 bytes

    Even though the files gain only a little by the RLE compressions (RLE is bad on reallife pictures) its seem to really hurt further compression.

    I will try to find some 256 colors "drawn" pictures to test next.

  11. #41
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    Here is a bunch of hand drawn 256 colour graphics for you: http://www.zenadsl5706.zen.co.uk/256...ixel%20Gfx.rar

    Oh...just noticed the angel particle picture isn't 256 colours in that archive, so you can exclude that. Has some nudity(Just breasts) in them hope you don't mind, if you do i can provide plenty of other examples.
    Last edited by Intrinsic; 8th August 2008 at 03:43.

  12. #42
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    926
    Thanks
    58
    Thanked 116 Times in 93 Posts
    just another quick test with the pictures form Intrinsic (The angels pixtures was reduced 256 colros with dithering)

    this time it used BMP/RLE vs BMP/RGB


    BMP/RGB 7 pictures = 2.068.986 bytes
    7-zip ultra = 705.113 bytes
    WinRAR best = 780.812 bytes
    NanoZip -cO = 594.936 bytes

    BMP/RLE 7 pictures = 1.689.696 bytes
    7-zip ultra = 819.095 bytes
    WinRAR Best = 887.394 bytes
    NanoZip -cO = 728.619 bytes


    Once again it seems to really help on compression by removing the RLE encoding first

    --- edit ---
    Targa (.TGA) files can also containe RLE encoding, and are used in a couple of games
    Last edited by SvenBent; 8th August 2008 at 04:31.

  13. #43
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    631
    Thanks
    288
    Thanked 252 Times in 128 Posts
    Especially in the truecolor example there's much to save, yes. I always delayed RLE recompression because I didn't know if the ratio gain would be worth it. Worst cases for RLE (truecolor) can gain much compression, but not always as much as storing them as non-RLE and compressing them.
    When recompressing, Precomp needs to store the RLE lengths, too, if they're not optimal. So there is no problem if 1024 black pixels are stored as (1024,0), but they can also be stored (to gain speed or random access or just because the encoder is crap) as something "evil" (=unexpected) like (2,0),(3,0),(4,0),...,(43,0),(44,0),(35,0).
    For optimal coded images or some other special cases that can be prepared (i.e. no RLE at all even if it would be possible), everything is fine and we just have the image data and some little overhead, but in other cases we could have to store all the lengths, too. This would create almost the same file as the RLE-coded image, except that the image data and the RLE lengths get splitted, but this won't help compression that much.
    Anyway, I can't imagine any worst case scenario where recompression would hurt the ratio, so it's worth a try as it is relatively simple to implement.
    http://schnaader.info
    Damn kids. They're all alike.

  14. #44
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    926
    Thanks
    58
    Thanked 116 Times in 93 Posts
    Yes i kinda thougth about the non optimal RLE values being a problem.
    And my "experiments doesnt take that into acount, because i actually throw the RLE away with no regards for keeping it later.



    For the files without optimal RLE encoding we could mabye detect if there seem to be a max lengt of the RLE encoding. We could detect if the RLE "stream is" that is logner then x is actually encoding at x + new stream

    E.g 300 black pixels are 255,0 45,0 if there is alot of 255 + new value we migh just save the RLE_maxlengt = 255 and then throw every info about RLE encoding with value off 255 aways as well.


    Please note that I have not compression programming skill.
    but i could think the non optimla RLE encoder would use some small buffer to get speed gains and therefore have the same artificial max of RLE lengt throghout the entire file.

    if that is the thruth.
    Then most nonoptimla RLE encoded pictures would only need the maxlengt information and we could throw out all RLE information.


    -- edit --

    if its not true regarding the maxlengt of RLE encoded "streams"
    i might have another idea. but since I'm at work a better do what I'm paid for... will be back with #2 idea soon.
    just a quick tip: "RLE encoding the RLE information"
    Last edited by SvenBent; 8th August 2008 at 12:48.

  15. #45
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    926
    Thanks
    58
    Thanked 116 Times in 93 Posts
    Hmm some more testing reveals that removing RLE encoding hurts compression in 7-zip and Winrar.

    These 256c pictures contains some primitive dithering in gradient areas. but it also contains large block of the same colors which are perfect for RLE.. it was kinda these type of pictures i believe would be typical for RLE encoding pictures.
    its taking for some H game named Sentimental Shooting

    Anyway i tested TGA/RLE BMP/RLE BMP/RGB

    Code:
    BMP_RGB.7z   10,5 MB (11.084.364 bytes)
    BMP_RGB.rar  10,1 MB (10.595.547 bytes)
    
    BMP_RLE.7z   10,3 MB  (10.838.794 bytes)
    BMP_RLE.rar   9,67 MB (10.146.599 bytes)
    
    TGA_RLE.7z   10,4 MB  (10.934.653 bytes)
    TGA_RLE.rar   9,66 MB (10.139.548 bytes)
    RLE -> compression is better then RGB -> compression

  16. #46
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    357
    Thanks
    37
    Thanked 39 Times in 24 Posts
    another format to improve support is SKP (google sketchup) files as it contains raster+vector data .. the raster is pure "PNG" in which precomp does not recognize.

    i have attached few samples to check..

    i tested "fan.skp" with no luck in normal, -slow -brute mode .. all fail to identify the embedded PNG files with "bitmap ripper v1.08" at http://mark0.net/soft-bitmaprip-e.html i could extract these easily its 6.0+7.0+13.0 KB of 29 kb file so it takes most of that file.

    precomp says there is three "png" streams but does nothing..
    i hope you can improve "png" detection more.. thanks anyway
    Attached Files Attached Files

  17. #47
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    357
    Thanks
    37
    Thanked 39 Times in 24 Posts
    another format that precomp doesnot support is "window media player skins" *.wmz

    samples are available at every windows version just look under

    C:\Program Files\Windows Media Player\Skins

    it normal zip file but not recognizable by precomp..

    advzip can handle it with "advzip -z0 *.wmz"

  18. #48
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    631
    Thanks
    288
    Thanked 252 Times in 128 Posts
    Quote Originally Posted by maadjordan View Post
    another format to improve support is SKP (google sketchup) files as it contains raster+vector data .. the raster is pure "PNG" in which precomp does not recognize.

    precomp says there is three "png" streams but does nothing..
    i hope you can improve "png" detection more.. thanks anyway
    Quote Originally Posted by maadjordan View Post
    another format that precomp doesnot support is "window media player skins" *.wmz

    samples are available at every windows version just look under

    C:\Program Files\Windows Media Player\Skins

    it normal zip file but not recognizable by precomp..
    Yes, PNG/ZIP/zLib recompression has to be improved. Note that decompression always works correctly, it is just bit-to-bit-identity that can't be restored. This will hopefully go away if I parse zLib streams myself, but this will take some time and will definitely not change for the next two versions, sorry...
    http://schnaader.info
    Damn kids. They're all alike.

  19. #49
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    357
    Thanks
    37
    Thanked 39 Times in 24 Posts
    Quote Originally Posted by schnaader View Post
    Yes, PNG/ZIP/zLib recompression has to be improved. Note that decompression always works correctly, it is just bit-to-bit-identity that can't be restored. This will hopefully go away if I parse zLib streams myself, but this will take some time and will definitely not change for the next two versions, sorry...
    if lossy mode is introduce (as an option) it would great to me.. and i can wait for even three versions.. but please keep it in your todo list and be patient with me as i shall proceed with my precomp testing with other formats..

  20. #50
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    216
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Love Precomp, an awesome utility... But always wondered, is there any chance to see an option in the future of processing .cab (.msi) files? Would be just great!

  21. #51
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    631
    Thanks
    288
    Thanked 252 Times in 128 Posts
    Quote Originally Posted by nanoflooder View Post
    Love Precomp, an awesome utility... But always wondered, is there any chance to see an option in the future of processing .cab (.msi) files? Would be just great!
    Yes, I also thought about this. I just saw there is a CAB SDK including some libraries available from Microsoft. Perhaps this will just work on CAB files, I'll have a look.
    http://schnaader.info
    Damn kids. They're all alike.

  22. #52
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    357
    Thanks
    37
    Thanked 39 Times in 24 Posts
    Quote Originally Posted by schnaader View Post
    Yes, I also thought about this. I just saw there is a CAB SDK including some libraries available from Microsoft. Perhaps this will just work on CAB files, I'll have a look.
    i suggest this one http://www.cabextract.org.uk/libmspack/ as it support more than cab(lzx..) please take it into consideration..

    for other links use

    http://www.speakeasy.org/~russotto/chm/
    http://mateusz.free.fr/mscab/

  23. #53
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    216
    Thanks
    0
    Thanked 0 Times in 0 Posts
    There's also some non-MS formats of CAB, like InstallShield 5, 6, 7... Is it possible for Precomp to be able to extract them as well?

  24. #54
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    357
    Thanks
    37
    Thanked 39 Times in 24 Posts
    first installshield CAB is different format than MS CAB.
    second till today .. no way to expand or edit then restore the original data as the compression method still need more investigation..

    the only version i could restore is InstallShield v3.0 which has formats .z,.1,.lib and its an old version .. all with icomp.exe or i3comp.exe or i95comp.exe

    but look here for source isdec,i5comp,i6comp

    ftp://ftp.uni-hannover.de/pub/mirror/os2/win32os2/
    Last edited by maadjordan; 4th September 2008 at 17:44.

  25. #55
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    631
    Thanks
    288
    Thanked 252 Times in 128 Posts
    Quote Originally Posted by maadjordan View Post
    first installshield CAB is different format than MS CAB.
    second till today .. no way to expand or edit then restore the original data as the compression method still need more investigation..

    the only version i could restore is InstallShield v3.0 which has formats .z,.1,.lib and its an old version .. all with icomp.exe or i3comp.exe or i95comp.exe
    I'm not quite sure about this. I have experimented with various InstallShield/installation/CAB files and it seems that some of the files are not valid CABs, but only the header or some filename information is missing and is stored somewhere else and with some trial-and-error you can often decompress most of the data (or does this only work for older version?) Also, I only tried to decompress with WinRAR, perhaps this detects various formats which aren't the same.

    Anyway, I guess one thing is sure: CAB won't be the easiest to implement... and the above is just for decompression, recompressing it bit-to-bit-identical will be a mess....
    http://schnaader.info
    Damn kids. They're all alike.

  26. #56
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    216
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by schnaader View Post
    Anyway, I guess one thing is sure: CAB won't be the easiest to implement... and the above is just for decompression, recompressing it bit-to-bit-identical will be a mess....
    Yep, that's 100% true... I haven't seen any app yet to do that Apart from maybe some $999.99 stuff like "Setup Squeezer", which apparently usually don't work at all because of their super-duper technology of compressing all files into one 7z SFX

  27. #57
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    357
    Thanks
    37
    Thanked 39 Times in 24 Posts
    i have to withdraw my previous comment.. installshield is already supported with precomp in "brute" mode so on large files in needs to be improved.. is seems the compression is based on deflate in some how .. i'v attached an example of v5.0
    Attached Files Attached Files

  28. #58
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    631
    Thanks
    288
    Thanked 252 Times in 128 Posts
    Quote Originally Posted by maadjordan View Post
    i have to withdraw my previous comment.. installshield is already supported with precomp in "brute" mode so on large files in needs to be improved.. is seems the compression is based on deflate in some how .. i'v attached an example of v5.0
    That's very interesting, especially as WinRAR fails to extract that CAB. I downloaded i5comp and it asks for _sys1.hdr. If there is such a file, could you upload it, too?
    http://schnaader.info
    Damn kids. They're all alike.

  29. #59
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    357
    Thanks
    37
    Thanked 39 Times in 24 Posts
    here is a full package

    ftp://ftp.net.pulawy.pl/pub/drivers/...D/APPLICATION/

    here is a topic from "i6comp.txt" inside "i6comp020.zip"

    1) No more ZDxxx.DLL - compression/decompression (deflate) routines are
    statically linked in.

    2)..

    3) All IShield engine files are now stuck into data1.cab, so convertion to
    single-volume cabinet has been changed. Conversion leaves data1.cab intact
    and gathers everything else into data2.cab.
    Data1.cab contains all installation and uninstallation files and
    is used by Uninstaller, and data2.cab contains all 'user' files.

    and take alook at zdata.lib i found "deflate Copyright 1995 Jean-loup Gailly"

    .. it confirms thats its zlib and data1.cab should be same of same version for all ..

    for data1.cab and data2.cab it works but in -brute mode only.. i tested http://www.traction-software.co.uk/d...chPrintPro.zip
    Last edited by maadjordan; 5th September 2008 at 03:49.

  30. #60
    Member
    Join Date
    Aug 2008
    Location
    Saint Petersburg, Russia
    Posts
    216
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Interesting thing about the InstallShield CABs - Precomp does indeed unpack them in -brute mode, and that's amazing, considering it cannot unpack the standard Microsoft CAB files, I have just checked this... So maybe it's just the MS CAB recompression technology that needs to be implemented inside Precomp?..

Page 2 of 4 FirstFirst 1234 LastLast

Similar Threads

  1. Precomp 0.4
    By schnaader in forum Data Compression
    Replies: 190
    Last Post: 5th October 2010, 16:13
  2. Precomp (and Precomp Comfort) in 315 kb
    By Yuri Grille. in forum Data Compression
    Replies: 2
    Last Post: 1st April 2009, 20:40
  3. Precomp 0.3.5 is out!
    By squxe in forum Forum Archive
    Replies: 1
    Last Post: 20th August 2007, 15:55
  4. Precomp 0.3.3 is out!
    By squxe in forum Forum Archive
    Replies: 1
    Last Post: 20th July 2007, 18:27

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •