Precomp 0.3.8

[maadjordan]

precomp still can't work with cab even the mszip which is a deflate algorithem.. so it needs much work to support the ms-cab

further more

another formats that is easy to support are AMV Video format based on the link http://code.google.com/p/amv-codec-t...vDocumentation its a mjpeg with Video format is jpeg (without DQT,SOF,SOS)..

samples are here http://www.mympxplayer.org/medium-dc11.html

thanks for precomp.. it seems i'm precomping my life..

btw for video samples look here

http://samples.mplayerhq.hu/V-codecs/

i've tested

mszh-zlib (-slow mode works..)

i have trubble using the -brute mode as its soooo slow .. i need to to speedup the brute mode..

finally .. another sucess is the "thumbs.db" in which precomp act very well.. i haven't test it with vista's thumbs version but i think it will work..

[maadjordan]

now an essential process is required is to decode encode files before using precomp and i think it can be done with precomp itself..

why to we need it..?
becuase different formats has embedded graphics in "encoded" mode. Currently (eml, mht,..) in which precomp does not recognize the embedded graphics..

samples for EML format can be downloaded here http://stationeryheaven.com/lund.htm

you can refer to wiki link here

http://en.wikipedia.org/wiki/Base64

[schnaader]

now an essential process is required is to decode encode files before using precomp and i think it can be done with precomp itself..

samples for EML format can be downloaded here http://stationeryheaven.com/lund.htm

100 points for you - you just guessed the new feature I am currently implementing for the next Precomp version. Results for one of the sample EML url:

arabianjewels_shlundeml.eml 240,399 bytes
arabianjewels_shlundeml.eml.ccm2 128.648 bytes
arabianjewels_shlundeml.pcf 177.030 bytes (2/2 Base64 streams)
arabianjewels_shlundeml.pcf.ccm2 119.479 bytes
arabianjewels_shlundeml_pcf.pcf 151.857 bytes (1/1 JPG stream)
arabianjewels_shlundeml_pcf.pcf.ccm2 102.496 bytes

So it's a ~20% gain for CCM 2, could be better for PAQ. Another interesting thing for PAQ will be that processing the decoded file should actually be faster as it is smaller
Also notice that for Base64, automatic multi-pass really gets an important item on my to-do list.

Of course, processing for example a big Thunderbird inbox with PDFs/JPGs/ZIPs inside should provide some better results

[maadjordan]

100 points for you - you just guessed the new feature I am currently implementing for the next Precomp version. Results for one of the sample EML url:

arabianjewels_shlundeml.eml 240,399 bytes
arabianjewels_shlundeml.eml.ccm2 128.648 bytes
arabianjewels_shlundeml.pcf 177.030 bytes (2/2 Base64 streams)
arabianjewels_shlundeml.pcf.ccm2 119.479 bytes
arabianjewels_shlundeml_pcf.pcf 151.857 bytes (1/1 JPG stream)
arabianjewels_shlundeml_pcf.pcf.ccm2 102.496 bytes

So it's a ~20% gain for CCM 2, could be better for PAQ. Another interesting thing for PAQ will be that processing the decoded file should actually be faster as it is smaller
Also notice that for Base64, automatic multi-pass really gets an important item on my to-do list.

Of course, processing for example a big Thunderbird inbox with PDFs/JPGs/ZIPs inside should provide some better results

great so base64 is under testing.. what about other encodings like

1- "yenc" its widly used now http://en.wikipedia.org/wiki/YEnc
2- "BinHex" main encoding under Mac OS http://en.wikipedia.org/wiki/BinHex
3- "Ascii85" & "ASCIIHex" inside PDF http://en.wikipedia.org/wiki/Ascii85
4- "uuencode" famous under unix http://en.wikipedia.org/wiki/Uuencoding

and yes automatic multi-pass is highly required

about pdf, does precomp unpack "CCITTFax" & "RunLength" ?

[maadjordan]

for examples refer to http://www.winzip.com/uuencode.htm
for dll that supports part of them refer to http://www.fpx.de/fp/Software/UUDeview/

finally this too http://www.kmonos.net/lib/xacrett.en.html

[SvenBent]

Leeloo: Leeloo Dallas mul-ti-pass.
Korben Dallas: Yeah.
Leeloo: Mul-ti-pass.
Korben Dallas: Yeah, multipass, she knows it's a multipass. Leeloo Dallas. This is my wife.
Leeloo: Mul-ti-pass.
Korben Dallas: We're newlyweds. Just met. You know how it is. We bumped into each other, sparks happen...
Leeloo: Mul-ti-pass.
Korben Dallas: Yes, she knows it's a multipass. Anyway, we're in love.



Sorry just couldn't resist it

[schnaader]

1- "yenc" its widly used now http://en.wikipedia.org/wiki/YEnc
2- "BinHex" main encoding under Mac OS http://en.wikipedia.org/wiki/BinHex
3- "Ascii85" & "ASCIIHex" inside PDF http://en.wikipedia.org/wiki/Ascii85
4- "uuencode" famous under unix http://en.wikipedia.org/wiki/Uuencoding

I'll try to add those too. Ascii85/AsciiHex was on my list as there are many
PDFs using it (often together with FlateDecode). Additionally, LZWEncode
should get supported.

about pdf, does precomp unpack "CCITTFax" & "RunLength" ?

Not yet, and this is definitely a lot trickier than the encodings above.
Perhaps I'll get this done when Precomp has gone beta (v0.5 and above).
Another PDF thing not supported yet are Deflate parameters and predictors
(like they are used in PNG).

[maadjordan]

thanks.. so when 0.39 is coming?

[schnaader]

thanks.. so when 0.39 is coming?

I'm currently moving into a new flat, so it definitely won't come out this month, I expect it to be finished in the middle of next month or a bit later if I decide to put some more features in.

[maadjordan]

another format to support is FlashPix it has JPEG streams that is not recognized by precomp 0.38

specifications http://graphcomp.com/info/specs/livepicture/fpx.pdf
samples (attached)..

another similar formats is Mix, (from ms pictureit)

then you have MDI (Microsoft Document Imaging Format) intruduced in office 2003, 2007.. (sample attached)..its processed but didnot check other modes..
http://office.microsoft.com/en-us/he...193601033.aspx

any new about v0.39 and your new flat

[schnaader]

another format to support is FlashPix it has JPEG streams that is not recognized by precomp 0.38

specifications http://graphcomp.com/info/specs/livepicture/fpx.pdf
samples (attached)..

another similar formats is Mix, (from ms pictureit)

then you have MDI (Microsoft Document Imaging Format) intruduced in office 2003, 2007.. (sample attached)..its processed but didnot check other modes..
http://office.microsoft.com/en-us/he...193601033.aspx

FlashPix seems to use some abbreviated JPEG format, perhaps some JPG headers can be added to complete it and to make PackJPG recognize it like with MJPEG. I will also have a look if other streams than JPG can appear in MDI.

any new about v0.39 and your new flat

We're currently renovating and there is more to do than we thought, so it will take some more time, but we already have a fresh green kitchen (no, not that bright, but it's a similar color)

[pat357]

I moved this post to here ; the original post by "Schneider" is here http://encode.su/forum/showpost.php?...&postcount=362

Have you tried using Precomp? I haven't had a look at the image, but if it contains some .gz files, it could help a bit.

Precomp seems to have a problem with this file : it stops at 47.9% processing and then hangs.
I left it running for 2h, but is still at 47.9%.
The latest time changes from the created "~temp0000000xx.dat" was 90 min ago.
Any clues ?
To be sure I stopped it and started all over again : same problem.

-- Edit : if you would like to test it youself :
source data is 4gb virtual machine image downloaded from http://www.noideashere.com/2008/08/gentoo-2008-vmware/
I used 7z to create a single .tar.

[schnaader]

Precomp seems to have a problem with this file : it stops at 47.9% processing and then hangs.
I left it running for 2h, but is still at 47.9%.
The latest time changes from the created "~temp0000000xx.dat" was 90 min ago.
Any clues ?
To be sure I stopped it and started all over again : same problem.

There is a bug in Precomp 0.38 speed for files bigger than 4 GB. It slows down (massively, it seems) at 4 GB - the PCF file is correct, however. But if it "stops" at 47,9% this would mean your test file would be > 8 GB... so perhaps it gets slower at 2 GB...
Anyway, I fixed the speed bug in the next Precomp version which should get available next month.
Until then you can try to split the file up into several smaller files (~1 GB), run Precomp over each of them and then combining them into a .tar file again.

[pat357]

There is a bug in Precomp 0.38 speed for files bigger than 4 GB. It slows down (massively, it seems) at 4 GB - the PCF file is correct, however. But if it "stops" at 47,9% this would mean your test file would be > 8 GB... so perhaps it gets slower at 2 GB...
Anyway, I fixed the speed bug in the next Precomp version which should get available next month.
Until then you can try to split the file up into several smaller files (~1 GB), run Precomp over each of them and then combining them into a .tar file again.

I used "rar -m0" to cut it in 1GB part.
Precomp now crashes on the 3e part (yes, that's were the 48,9% was).
I tried to find to segment where precomp crashed on by further splitting.
The smallest I could get while still crashing is about 200MB uncompressed (21MB 7z compressed).

http://rapidshare.com/files/156544093/gentoo_spit.7z (21 MB)

Could you please have a look at it ?

[schnaader]

I used "rar -m0" to cut it in 1GB part.
Precomp now crashes on the 3e part (yes, that's were the 48,9% was).
I tried to find to segment where precomp crashed on by further splitting.
The smallest I could get while still crashing is about 200MB uncompressed (21MB 7z compressed).

http://rapidshare.com/files/156544093/gentoo_spit.7z (21 MB)

Could you please have a look at it ?

There happens to be a misdetected JPG at position 72727977 of this file with a size of 136 MB, so I guess there is a start marker, but no end marker was found. Detecting and processing JPGs is quite slow and the "size" of the JPG will grow for bigger files. There's no crash here, by the way, it's just very slow (105 seconds versus 40 s without JPG processing).
For this part, there are no other JPG streams, so you can switch JPG processing off using "-t-j" as a switch.
You can also ignore this special JPG stream by first running debug mode ("-v") and then ignoring it with the "-i" switch.

Seems like I should improve JPG detection speed and perhaps add a switch to set max. JPG size (a 100 MB JPG is very unlikely...)

[m^2]

Precomp crashes on this file.
I failed to split it and preserve the problem.

[maadjordan]

Precomp crashes on this file.
I failed to split it and preserve the problem.

if you have 7-zip use file>split_files its better than downloading 100mb..

...

any news about precomp coming version..?

[m^2]

if you have 7-zip use file>split_files its better than downloading 100mb..

...

any news about precomp coming version..?

I meant that however I split it, the problem was gone.

[Simon Berger]

For every crash try if it also happens if you disable jpg compression. It has a bug on some pictures which are big and with a wrong data.
I hope it will be fixed any time.
The bug appears later because packjpg only creates memory corruption so it is hard to find with segmentation.

[m^2]

For every crash try if it also happens if you disable jpg compression. It has a bug on some pictures which are big and with a wrong data.
I hope it will be fixed any time.
The bug appears later because packjpg only creates memory corruption so it is hard to find with segmentation.

Edited.
OK, it seems to be it.

[schnaader]

any news about precomp coming version..?

I'm currently doing some tests with Base64 and bZip. Base64 is almost done, but there are some special cases that have to be refined, and bZip is the reason why there is no new version yet as it will be a very useful feature and is easier to implement than zLib, but still needs a little time.

Precomp crashes on this file.
I failed to split it and preserve the problem.

Hmm... I could only recreate the error sometimes and when I could, it happened at a GIF file close to the end of the file, but as Simon Berger said, it's not the GIF file, but seems to be some other file that causes some memory error. I don't know if it is packJPG though, as my test results were very confusing, so I'll keep an eye on it.

By the way, your .arc file was confusing me a bit. It seems to contain the crash file named "crash" and also a directory named "crash" that contains some other files (looks like a Total Commander installation). FreeArc extracts some of the files in the directory first and then fails to create the file because it has the same name as the directory. I handled this by renaming the directory when FreeArc shows the overwrite dialog.

[m^2]

Hmm... I could only recreate the error sometimes and when I could, it happened at a GIF file close to the end of the file, but as Simon Berger said, it's not the GIF file, but seems to be some other file that causes some memory error. I don't know if it is packJPG though, as my test results were very confusing, so I'll keep an eye on it.

By the way, your .arc file was confusing me a bit. It seems to contain the crash file named "crash" and also a directory named "crash" that contains some other files (looks like a Total Commander installation). FreeArc extracts some of the files in the directory first and then fails to create the file because it has the same name as the directory. I handled this by renaming the directory when FreeArc shows the overwrite dialog.

Oops, I mistakingly added some random files. I don't know when, how, ...
The main file is "crash", you can forget about the rest.
I've had some random crashes with FreeArc -max on a related file set, but they were not reproducible, with this file precomp always crashes on my machine.

[maadjordan]

I'm currently doing some tests with Base64 and bZip. Base64 is almost done, but there are some special cases that have to be refined, and bZip is the reason why there is no new version yet as it will be a very useful feature and is easier to implement than zLib, but still needs a little time.

so supporting pdf made by Multivalent (with bzip2 filter BZip2Decode) http://multivalent.sourceforge.net/R...ompactPDF.html is in progress

[kaitz]

Is there any news/progress?

[DeathTheSheep]

schnaader:
Perhaps if you are no longer interested in maintaining this project, you could release the source code in its current form so others can? Just a suggestion.

[Intrinsic]

I think i read in another thread he is still working on it, but this is a hobby project remember so RL stuff gets in the way From memory he mentioned 0.3.9 being pretty close to release.

[schnaader]

I think i read in another thread he is still working on it, but this is a hobby project remember so RL stuff gets in the way From memory he mentioned 0.3.9 being pretty close to release.

Absolutely right. In fact, the next release will be 0.4 because there are many new features and I think it's better to release this version instead of a 0.3.9 version that is only used for a month. But I still don't know when the exact release date will be.

[DeathTheSheep]

That's excellent news, then! (Now awaiting 0.4! )

[SvenBent]

i now this i way out of what precomp is doing.

but maybe precomp could extra audio tracks from cue/bin files to separate files. and remove the data from the .bin file.

that way compression programs can maybe better detect the audio part at gain better compressions.

so instead of
.cue (txt)
.bin (data + audio + EEC)

we could have
.cue (txt)
.bin (only data parts + ecm)
.1.wav -> .99.wav (audio tracks)
*. pcf (info to properly put the audio back into the .bin file)

maybe even apply ECM filtering on the .bin data

[maadjordan]

Absolutely right. In fact, the next release will be 0.4 because there are many new features and I think it's better to release this version instead of a 0.3.9 version that is only used for a month. But I still don't know when the exact release date will be.

great .. more patience that i can do.. no problem.