Page 3 of 4 FirstFirst 1234 LastLast
Results 61 to 90 of 117

Thread: Precomp 0.3.8

  1. #61
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    349
    Thanks
    37
    Thanked 37 Times in 22 Posts
    precomp still can't work with cab even the mszip which is a deflate algorithem.. so it needs much work to support the ms-cab

    further more

    another formats that is easy to support are AMV Video format based on the link http://code.google.com/p/amv-codec-t...vDocumentation its a mjpeg with Video format is jpeg (without DQT,SOF,SOS)..

    samples are here http://www.mympxplayer.org/medium-dc11.html

    thanks for precomp.. it seems i'm precomping my life..

    btw for video samples look here

    http://samples.mplayerhq.hu/V-codecs/

    i've tested

    mszh-zlib (-slow mode works..)

    i have trubble using the -brute mode as its soooo slow .. i need to to speedup the brute mode..

    finally .. another sucess is the "thumbs.db" in which precomp act very well.. i haven't test it with vista's thumbs version but i think it will work..
    Last edited by maadjordan; 18th September 2008 at 22:58.

  2. #62
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    349
    Thanks
    37
    Thanked 37 Times in 22 Posts
    now an essential process is required is to decode encode files before using precomp and i think it can be done with precomp itself..

    why to we need it..?
    becuase different formats has embedded graphics in "encoded" mode. Currently (eml, mht,..) in which precomp does not recognize the embedded graphics..

    samples for EML format can be downloaded here http://stationeryheaven.com/lund.htm

    you can refer to wiki link here

    http://en.wikipedia.org/wiki/Base64
    Last edited by maadjordan; 20th September 2008 at 15:25.

  3. #63
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    612
    Thanks
    250
    Thanked 240 Times in 119 Posts
    Quote Originally Posted by maadjordan View Post
    now an essential process is required is to decode encode files before using precomp and i think it can be done with precomp itself..

    samples for EML format can be downloaded here http://stationeryheaven.com/lund.htm
    100 points for you - you just guessed the new feature I am currently implementing for the next Precomp version. Results for one of the sample EML url:

    arabianjewels_shlundeml.eml 240,399 bytes
    arabianjewels_shlundeml.eml.ccm2 128.648 bytes
    arabianjewels_shlundeml.pcf 177.030 bytes (2/2 Base64 streams)
    arabianjewels_shlundeml.pcf.ccm2 119.479 bytes
    arabianjewels_shlundeml_pcf.pcf 151.857 bytes (1/1 JPG stream)
    arabianjewels_shlundeml_pcf.pcf.ccm2 102.496 bytes

    So it's a ~20% gain for CCM 2, could be better for PAQ. Another interesting thing for PAQ will be that processing the decoded file should actually be faster as it is smaller
    Also notice that for Base64, automatic multi-pass really gets an important item on my to-do list.

    Of course, processing for example a big Thunderbird inbox with PDFs/JPGs/ZIPs inside should provide some better results
    http://schnaader.info
    Damn kids. They're all alike.

  4. #64
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    349
    Thanks
    37
    Thanked 37 Times in 22 Posts
    Quote Originally Posted by schnaader View Post
    100 points for you - you just guessed the new feature I am currently implementing for the next Precomp version. Results for one of the sample EML url:

    arabianjewels_shlundeml.eml 240,399 bytes
    arabianjewels_shlundeml.eml.ccm2 128.648 bytes
    arabianjewels_shlundeml.pcf 177.030 bytes (2/2 Base64 streams)
    arabianjewels_shlundeml.pcf.ccm2 119.479 bytes
    arabianjewels_shlundeml_pcf.pcf 151.857 bytes (1/1 JPG stream)
    arabianjewels_shlundeml_pcf.pcf.ccm2 102.496 bytes

    So it's a ~20% gain for CCM 2, could be better for PAQ. Another interesting thing for PAQ will be that processing the decoded file should actually be faster as it is smaller
    Also notice that for Base64, automatic multi-pass really gets an important item on my to-do list.

    Of course, processing for example a big Thunderbird inbox with PDFs/JPGs/ZIPs inside should provide some better results
    great so base64 is under testing.. what about other encodings like

    1- "yenc" its widly used now http://en.wikipedia.org/wiki/YEnc
    2- "BinHex" main encoding under Mac OS http://en.wikipedia.org/wiki/BinHex
    3- "Ascii85" & "ASCIIHex" inside PDF http://en.wikipedia.org/wiki/Ascii85
    4- "uuencode" famous under unix http://en.wikipedia.org/wiki/Uuencoding

    and yes automatic multi-pass is highly required

    about pdf, does precomp unpack "CCITTFax" & "RunLength" ?
    Last edited by maadjordan; 20th September 2008 at 23:30.

  5. #65
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    349
    Thanks
    37
    Thanked 37 Times in 22 Posts
    for examples refer to http://www.winzip.com/uuencode.htm
    for dll that supports part of them refer to http://www.fpx.de/fp/Software/UUDeview/

    finally this too http://www.kmonos.net/lib/xacrett.en.html
    Last edited by maadjordan; 23rd September 2008 at 14:14.

  6. #66
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    918
    Thanks
    57
    Thanked 113 Times in 90 Posts
    Leeloo: Leeloo Dallas mul-ti-pass.
    Korben Dallas: Yeah.
    Leeloo: Mul-ti-pass.
    Korben Dallas: Yeah, multipass, she knows it's a multipass. Leeloo Dallas. This is my wife.
    Leeloo: Mul-ti-pass.
    Korben Dallas: We're newlyweds. Just met. You know how it is. We bumped into each other, sparks happen...
    Leeloo: Mul-ti-pass.
    Korben Dallas: Yes, she knows it's a multipass. Anyway, we're in love.



    Sorry just couldn't resist it

  7. #67
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    612
    Thanks
    250
    Thanked 240 Times in 119 Posts
    Quote Originally Posted by maadjordan View Post
    1- "yenc" its widly used now http://en.wikipedia.org/wiki/YEnc
    2- "BinHex" main encoding under Mac OS http://en.wikipedia.org/wiki/BinHex
    3- "Ascii85" & "ASCIIHex" inside PDF http://en.wikipedia.org/wiki/Ascii85
    4- "uuencode" famous under unix http://en.wikipedia.org/wiki/Uuencoding
    I'll try to add those too. Ascii85/AsciiHex was on my list as there are many
    PDFs using it (often together with FlateDecode). Additionally, LZWEncode
    should get supported.

    Quote Originally Posted by maadjordan View Post
    about pdf, does precomp unpack "CCITTFax" & "RunLength" ?
    Not yet, and this is definitely a lot trickier than the encodings above.
    Perhaps I'll get this done when Precomp has gone beta (v0.5 and above).
    Another PDF thing not supported yet are Deflate parameters and predictors
    (like they are used in PNG).
    http://schnaader.info
    Damn kids. They're all alike.

  8. #68
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    349
    Thanks
    37
    Thanked 37 Times in 22 Posts
    thanks.. so when 0.39 is coming?

  9. #69
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    612
    Thanks
    250
    Thanked 240 Times in 119 Posts
    Quote Originally Posted by maadjordan View Post
    thanks.. so when 0.39 is coming?
    I'm currently moving into a new flat, so it definitely won't come out this month, I expect it to be finished in the middle of next month or a bit later if I decide to put some more features in.
    http://schnaader.info
    Damn kids. They're all alike.

  10. #70
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    349
    Thanks
    37
    Thanked 37 Times in 22 Posts

    Talking

    another format to support is FlashPix it has JPEG streams that is not recognized by precomp 0.38

    specifications http://graphcomp.com/info/specs/livepicture/fpx.pdf
    samples (attached)..

    another similar formats is Mix, (from ms pictureit)

    then you have MDI (Microsoft Document Imaging Format) intruduced in office 2003, 2007.. (sample attached)..its processed but didnot check other modes..
    http://office.microsoft.com/en-us/he...193601033.aspx

    any new about v0.39 and your new flat
    Attached Files Attached Files

  11. #71
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    612
    Thanks
    250
    Thanked 240 Times in 119 Posts
    Quote Originally Posted by maadjordan View Post
    another format to support is FlashPix it has JPEG streams that is not recognized by precomp 0.38

    specifications http://graphcomp.com/info/specs/livepicture/fpx.pdf
    samples (attached)..

    another similar formats is Mix, (from ms pictureit)

    then you have MDI (Microsoft Document Imaging Format) intruduced in office 2003, 2007.. (sample attached)..its processed but didnot check other modes..
    http://office.microsoft.com/en-us/he...193601033.aspx
    FlashPix seems to use some abbreviated JPEG format, perhaps some JPG headers can be added to complete it and to make PackJPG recognize it like with MJPEG. I will also have a look if other streams than JPG can appear in MDI.

    Quote Originally Posted by maadjordan View Post
    any new about v0.39 and your new flat
    We're currently renovating and there is more to do than we thought, so it will take some more time, but we already have a fresh green kitchen (no, not that bright, but it's a similar color)
    http://schnaader.info
    Damn kids. They're all alike.

  12. #72
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts

    Arrow Precomp stops processing after 47.9%

    I moved this post to here ; the original post by "Schneider" is here http://encode.su/forum/showpost.php?...&postcount=362

    Quote Originally Posted by schnaader View Post
    Have you tried using Precomp? I haven't had a look at the image, but if it contains some .gz files, it could help a bit.
    Precomp seems to have a problem with this file : it stops at 47.9% processing and then hangs.
    I left it running for 2h, but is still at 47.9%.
    The latest time changes from the created "~temp0000000xx.dat" was 90 min ago.
    Any clues ?
    To be sure I stopped it and started all over again : same problem.

    -- Edit : if you would like to test it youself :
    source data is 4gb virtual machine image downloaded from http://www.noideashere.com/2008/08/gentoo-2008-vmware/
    I used 7z to create a single .tar.

  13. #73
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    612
    Thanks
    250
    Thanked 240 Times in 119 Posts
    Quote Originally Posted by pat357 View Post
    Precomp seems to have a problem with this file : it stops at 47.9% processing and then hangs.
    I left it running for 2h, but is still at 47.9%.
    The latest time changes from the created "~temp0000000xx.dat" was 90 min ago.
    Any clues ?
    To be sure I stopped it and started all over again : same problem.
    There is a bug in Precomp 0.38 speed for files bigger than 4 GB. It slows down (massively, it seems) at 4 GB - the PCF file is correct, however. But if it "stops" at 47,9% this would mean your test file would be > 8 GB... so perhaps it gets slower at 2 GB...
    Anyway, I fixed the speed bug in the next Precomp version which should get available next month.
    Until then you can try to split the file up into several smaller files (~1 GB), run Precomp over each of them and then combining them into a .tar file again.
    http://schnaader.info
    Damn kids. They're all alike.

  14. #74
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by schnaader View Post
    There is a bug in Precomp 0.38 speed for files bigger than 4 GB. It slows down (massively, it seems) at 4 GB - the PCF file is correct, however. But if it "stops" at 47,9% this would mean your test file would be > 8 GB... so perhaps it gets slower at 2 GB...
    Anyway, I fixed the speed bug in the next Precomp version which should get available next month.
    Until then you can try to split the file up into several smaller files (~1 GB), run Precomp over each of them and then combining them into a .tar file again.
    I used "rar -m0" to cut it in 1GB part.
    Precomp now crashes on the 3e part (yes, that's were the 48,9% was).
    I tried to find to segment where precomp crashed on by further splitting.
    The smallest I could get while still crashing is about 200MB uncompressed (21MB 7z compressed).

    http://rapidshare.com/files/156544093/gentoo_spit.7z (21 MB)

    Could you please have a look at it ?

  15. #75
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    612
    Thanks
    250
    Thanked 240 Times in 119 Posts
    Quote Originally Posted by pat357 View Post
    I used "rar -m0" to cut it in 1GB part.
    Precomp now crashes on the 3e part (yes, that's were the 48,9% was).
    I tried to find to segment where precomp crashed on by further splitting.
    The smallest I could get while still crashing is about 200MB uncompressed (21MB 7z compressed).

    http://rapidshare.com/files/156544093/gentoo_spit.7z (21 MB)

    Could you please have a look at it ?
    There happens to be a misdetected JPG at position 72727977 of this file with a size of 136 MB, so I guess there is a start marker, but no end marker was found. Detecting and processing JPGs is quite slow and the "size" of the JPG will grow for bigger files. There's no crash here, by the way, it's just very slow (105 seconds versus 40 s without JPG processing).
    For this part, there are no other JPG streams, so you can switch JPG processing off using "-t-j" as a switch.
    You can also ignore this special JPG stream by first running debug mode ("-v") and then ignoring it with the "-i" switch.

    Seems like I should improve JPG detection speed and perhaps add a switch to set max. JPG size (a 100 MB JPG is very unlikely...)
    http://schnaader.info
    Damn kids. They're all alike.

  16. #76
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Precomp crashes on this file.
    I failed to split it and preserve the problem.

  17. #77
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    349
    Thanks
    37
    Thanked 37 Times in 22 Posts
    Quote Originally Posted by m^2 View Post
    Precomp crashes on this file.
    I failed to split it and preserve the problem.
    if you have 7-zip use file>split_files its better than downloading 100mb..

    ...

    any news about precomp coming version..?

  18. #78
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by maadjordan View Post
    if you have 7-zip use file>split_files its better than downloading 100mb..

    ...

    any news about precomp coming version..?
    I meant that however I split it, the problem was gone.

  19. #79
    Member
    Join Date
    Oct 2007
    Location
    Germany, Hamburg
    Posts
    409
    Thanks
    0
    Thanked 5 Times in 5 Posts
    For every crash try if it also happens if you disable jpg compression. It has a bug on some pictures which are big and with a wrong data.
    I hope it will be fixed any time.
    The bug appears later because packjpg only creates memory corruption so it is hard to find with segmentation.

  20. #80
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Simon Berger View Post
    For every crash try if it also happens if you disable jpg compression. It has a bug on some pictures which are big and with a wrong data.
    I hope it will be fixed any time.
    The bug appears later because packjpg only creates memory corruption so it is hard to find with segmentation.
    Edited.
    OK, it seems to be it.

  21. #81
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    612
    Thanks
    250
    Thanked 240 Times in 119 Posts
    Quote Originally Posted by maadjordan View Post
    any news about precomp coming version..?
    I'm currently doing some tests with Base64 and bZip. Base64 is almost done, but there are some special cases that have to be refined, and bZip is the reason why there is no new version yet as it will be a very useful feature and is easier to implement than zLib, but still needs a little time.

    Quote Originally Posted by m^2 View Post
    Precomp crashes on this file.
    I failed to split it and preserve the problem.
    Hmm... I could only recreate the error sometimes and when I could, it happened at a GIF file close to the end of the file, but as Simon Berger said, it's not the GIF file, but seems to be some other file that causes some memory error. I don't know if it is packJPG though, as my test results were very confusing, so I'll keep an eye on it.

    By the way, your .arc file was confusing me a bit. It seems to contain the crash file named "crash" and also a directory named "crash" that contains some other files (looks like a Total Commander installation). FreeArc extracts some of the files in the directory first and then fails to create the file because it has the same name as the directory. I handled this by renaming the directory when FreeArc shows the overwrite dialog.
    http://schnaader.info
    Damn kids. They're all alike.

  22. #82
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by schnaader View Post
    Hmm... I could only recreate the error sometimes and when I could, it happened at a GIF file close to the end of the file, but as Simon Berger said, it's not the GIF file, but seems to be some other file that causes some memory error. I don't know if it is packJPG though, as my test results were very confusing, so I'll keep an eye on it.

    By the way, your .arc file was confusing me a bit. It seems to contain the crash file named "crash" and also a directory named "crash" that contains some other files (looks like a Total Commander installation). FreeArc extracts some of the files in the directory first and then fails to create the file because it has the same name as the directory. I handled this by renaming the directory when FreeArc shows the overwrite dialog.
    Oops, I mistakingly added some random files. I don't know when, how, ...
    The main file is "crash", you can forget about the rest.
    I've had some random crashes with FreeArc -max on a related file set, but they were not reproducible, with this file precomp always crashes on my machine.

  23. #83
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    349
    Thanks
    37
    Thanked 37 Times in 22 Posts
    Quote Originally Posted by schnaader View Post
    I'm currently doing some tests with Base64 and bZip. Base64 is almost done, but there are some special cases that have to be refined, and bZip is the reason why there is no new version yet as it will be a very useful feature and is easier to implement than zLib, but still needs a little time.
    so supporting pdf made by Multivalent (with bzip2 filter BZip2Decode) http://multivalent.sourceforge.net/R...ompactPDF.html is in progress

  24. #84
    Member
    Join Date
    May 2008
    Location
    Estonia
    Posts
    509
    Thanks
    208
    Thanked 347 Times in 185 Posts
    Is there any news/progress?

    KZo


  25. #85
    Member
    Join Date
    Dec 2008
    Posts
    12
    Thanks
    0
    Thanked 0 Times in 0 Posts
    schnaader:
    Perhaps if you are no longer interested in maintaining this project, you could release the source code in its current form so others can? Just a suggestion.

  26. #86
    Member
    Join Date
    May 2008
    Location
    England
    Posts
    325
    Thanks
    18
    Thanked 6 Times in 5 Posts
    I think i read in another thread he is still working on it, but this is a hobby project remember so RL stuff gets in the way From memory he mentioned 0.3.9 being pretty close to release.

  27. #87
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    612
    Thanks
    250
    Thanked 240 Times in 119 Posts
    Quote Originally Posted by Intrinsic View Post
    I think i read in another thread he is still working on it, but this is a hobby project remember so RL stuff gets in the way From memory he mentioned 0.3.9 being pretty close to release.
    Absolutely right. In fact, the next release will be 0.4 because there are many new features and I think it's better to release this version instead of a 0.3.9 version that is only used for a month. But I still don't know when the exact release date will be.
    http://schnaader.info
    Damn kids. They're all alike.

  28. #88
    Member
    Join Date
    Dec 2008
    Posts
    12
    Thanks
    0
    Thanked 0 Times in 0 Posts
    That's excellent news, then! (Now awaiting 0.4! )

  29. #89
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    918
    Thanks
    57
    Thanked 113 Times in 90 Posts
    i now this i way out of what precomp is doing.

    but maybe precomp could extra audio tracks from cue/bin files to separate files. and remove the data from the .bin file.

    that way compression programs can maybe better detect the audio part at gain better compressions.

    so instead of
    .cue (txt)
    .bin (data + audio + EEC)

    we could have
    .cue (txt)
    .bin (only data parts + ecm)
    .1.wav -> .99.wav (audio tracks)
    *. pcf (info to properly put the audio back into the .bin file)

    maybe even apply ECM filtering on the .bin data
    Last edited by SvenBent; 29th December 2008 at 19:47.

  30. #90
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    349
    Thanks
    37
    Thanked 37 Times in 22 Posts
    Quote Originally Posted by schnaader View Post
    Absolutely right. In fact, the next release will be 0.4 because there are many new features and I think it's better to release this version instead of a 0.3.9 version that is only used for a month. But I still don't know when the exact release date will be.

    great .. more patience that i can do.. no problem.

Page 3 of 4 FirstFirst 1234 LastLast

Similar Threads

  1. Precomp 0.4
    By schnaader in forum Data Compression
    Replies: 190
    Last Post: 5th October 2010, 15:13
  2. Precomp (and Precomp Comfort) in 315 kb
    By Yuri Grille. in forum Data Compression
    Replies: 2
    Last Post: 1st April 2009, 19:40
  3. Precomp 0.3.5 is out!
    By squxe in forum Forum Archive
    Replies: 1
    Last Post: 20th August 2007, 14:55
  4. Precomp 0.3.3 is out!
    By squxe in forum Forum Archive
    Replies: 1
    Last Post: 20th July 2007, 17:27

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •