Precomp source code on GitHub

[schnaader]

After a long time of abscence (main reason: I have a son now! ), I finally managed to upload the Precomp source code to GitHub: https://github.com/schnaader/precomp-cpp. I'll put some kind of roadmap there soon, fill the issue tracker and make some refactorings so that this can be a final 0.4.4 release (until now, it's marked as an "unstable" version).

I couldn't test the Linux version yet, so it would be nice if someone could try to run make.sh and report success or failure. Also, my g++ version for Windows is 4.8.1, so testing the Windows version would also be useful.

I've seen AntiZ and its progress, well done, Diazonium!

I'm looking forward to feedback on this, especially advice regarding license stuff, GitHub and possible ways to clean up and improve the code.

[comp1]

Wow this is great!

Thank you so much.

[Dimitri]

Very hard to compile on Microsoft Visual 2013, Can you provide some info please ??

i will try with dev-cpp as well

[schnaader]

Very hard to compile on Microsoft Visual 2013, Can you provide some info please ??

I've added an issue for it. Solving external dependencies (in this case, the "contrib" folder) can get quite confusing in MSVC and I have just recently started using Visual Studio for C/C++ projects. Will look at it after the weekend, but this is an issue that others could help with, too.

Speaking of, the issue tracker is now filled with some entries and there's a tag "low hanging fruits" where people should be able to help even if they aren't familiar with the project or the involved algorithms.

[Skymmer]

schnaader, its nice to see that you haven't disappeared in a void
My congratulations about a son! And thanks for providing the sources of Precomp

Here are the both x86 and x64 builds of PreComp 0.4.4 (20160106_deaabc1)

EDIT: Attachment removed. x64 version crashes on JPEG streams. Get the updated version below.

[schnaader]

Here are the both x86 and x64 builds of PreComp 0.4.4 (20160106_deaabc1)

Interesting, never thought about a 64-bit build, but from a quick test, it seems to be 5-15% faster on a 64-bit machine, reproducible. How did you compile? Using "-m64"? Have you modified any source for it?

[Skymmer]

Sources are quite perfect from the compilation point of view, so I made no modifications at all.
Yes, its just -m64 and changed -march=pentiumpro to -march=x86-64
Compiler: mingw64-4.9.3-rt_v4-rev1 (winthreads, seh)

[Dimitri]

i wanted to add the packjpg newest version, but i couldnt manage to get this compiled // will check again in a few days

this comes with packjpg 2.5a and i have sources from packjpg 2.5f many versions later are they even compatible ??

thanks in advance !!

Edit 1 ) precomp 64bit crashes !! i run the test twice just to make sure it crashed both times

[External compressorrecomp]
header = 0
packcmd = precomp -intense0 -o$$arcpackedfile$$.tmp $$arcdatafile$$.tmp
unpackcmd = precomp -o$$arcdatafile$$.tmp -r $$arcpackedfile$$.tmp

this is the method i used used as external compressor

[schnaader]

Very hard to compile on Microsoft Visual 2013, Can you provide some info please ??

I just added a Visual Studio 2012 solution to the project (path vs2012), this should be compatible with VS2013.
For reference: The commit that adds the solution.

[Dimitri]

I also saw that you are trying to make it Multithreaded, i am trying to figure out how a stdin and stdout can become a part of this process

https://mega.nz/#!mtwCxJyC!K2nlQi0H-9nC3xyJ139HrDrfB5j_yg83C5sKjXKizRg

[maadjordan]

I think that the best addition to precomp will be adding packArc as it will add packMP3 and packPNM immediately. it will collaborate efforts

[schnaader]

I think that the best addition to precomp will be adding packArc as it will add packMP3 and packPNM immediately. it will collaborate efforts

Latest development branch (commit 0bc5252) uses packMP3 to recompress MP3 files. ZIP file containing Windows and Linux executables attached to this post. It uses the same detection as packMP3 (five consecutive MP3 frames with same flags), but doesn't parse ID3 tags. This way, ID3 tags will be left untouched by packMP3 and Precomp can process them, e.g. JPG thumbnails inside them. MP3 detection is less restrictive than packMP3's detection and thus some MP3s can be processed that aren't successfully recompressed by packMP3, but this seems to be a quite rare case (e.g. for MP3s containing large ID3 tags).

There is some place for improvements, e.g. files not supported by packMP3 like most from here will take very long to process as Precomp detects a potential MP3 file at each frame and passes the remaining frames to packMP3.

Test with 63 files from austrian mashup producer DJ Schmolli inside a 7-Zip archive:

>precomp -cn -v DJ_Schmolli.7z

New size: 780995898 instead of 903465193

Time: 14 minute(s), 23 second(s)

Recompressed streams: 97/97
JPG streams: 31/31
JPG streams (progressive): 3/3
MP3 streams: 63/63

>precomp -r DJ_Schmolli.pcf

Time: 16 minute(s), 7 second(s)

The resulting file is bitwise identical to the original 7-Zip archive.

[Skymmer]

Edit 1 ) precomp 64bit crashes !! i run the test twice just to make sure it crashed both times

Yes, the x64 build I provided is really crashes. But its only happens on data with JPEG streams inside.
The reason is the PackJPG v2.5a which is not too much friendly to x64. But the good thing is that such behaviour have been fixed in later versions and since Precomp officially migrated to PackJPG v2.5j, I have updated the binaries and published them in Precomp 0.4.4 thread.

[Gonzalo]

Congrats for being father! So glad to see precomp still alive. Pioneer software, now including MP3 support, goes epic!
Don't forget about MP2 recompression... Very useful on video files.
Open source software, very fast processing, about 15% gaining on ratio.

unpackmp2_v1.2_.zip

Btw, I know this would be difficult, but... Is there any chance to include LZX recompression? (.msi instalers, .cab archivers, .chm help files, etc...) Don't know about patent status.

[Gonzalo]

Don't know about patent status.

R:

The purpose of this document is to allow anyone to encode or decode LZX compressed data.

msdn.microsoft.com/en-us/library/bb417343.aspx#lzxdatacompressionformat

[edcassa]

It seems that the latest commit destroyed the modifications made for Visual studio.

[schnaader]

Yes, the bitops.cpp modification got lost when updating to new packJPG. I fixed this in the refactoring branch (commit cc0ca6). Will merge it back to master branch soon.

[Dimitri]

MR. Schaader Hello , i try to work on a handle for stdin and stdout ....



GetStdHandle(STD_INPUT_HANDLE)


//stdin1
DWORD WINAPI GetFileSize(
_In_ HANDLE hFile,
_Out_opt_ LPDWORD lpFileSizeHigh
);


//fin_length = fileSize64(in_file); old

seems like i have to change a lot in order for precomp not to see a filename and a handler instead, can you help with this if you have the time ?

[schnaader]

Hi Dimitri,

don't use GetStdHandle, it will give you a Win32 HANDLE that is not compatible with FILE* (as you said). There are handles for stdin/stdout defined already by the standard library, you can just use them:

char c = fgetc(stdin);

Anyway, there are other things that have to be changed for stdin/stdout support:

1. stdin/stdout are streams, they don't give you a size.
2. You can't seek in stdin/stdout streams.
3. At the moment, all of Precomp's console output goes to stdout, this has to be changed to stderr.

Especially the first two are tricky and need some major changes, that's why I wrote this to the wishlist of Precomp v0.5 and haven't opened an issue about it yet. A somewhat easier subtask would be recompressing to stdout (like "precomp -r myfile.pcf > original.pdf", you'd "only" have to do number 3 for it and validate if the code complies to 1 and 2.

[Dimitri]

thanks for this, precomp can be made fast with stdout support, i use shar || srep || zstd for windows compression .... adding shar <stdin> <stdout> || precomp <stdin> <stdout> || srep <stdin> <stdout> || zstd <stdin> <stdout>

[Bulat Ziganshin]

i should note that in Windows stdin/stdout has sizes (and probably seekable) if they represent real files like in the command:

precomp <file1 >file2

but when they represent pipes to other commands, they of course lose such ability

[Gonzalo]

I found what might be a tiny bug:

Recompressed streams: 90/90
PDF streams: 88/88
PDF image streams (8-bit): 0/1
JPG streams: 1/1

88+0+1=89 ≠ 90
Consistent on various different files. Always the issue is related to PDF image streams miscounted.

---------------------------

I also observed two kinds of failures when processing mp3 files:
1) Valid MP3s aren't recognized at all.
2) Hundreds of MP3 streams found in a single valid file which aren't processed either.

I have examples of both. If you want them I can upload the files plus the verbose log.

---------------------------

Trying to speed up the things, I've been using ppx2 to make the batch multi-threaded. But different instances of precomp tries to write their temp files with the same exact name, leading to errors. So, in the meantime precomp still needing those temp files, you might make them unique by giving them a variable as a name, like a random number or <originalfile_########.dat>


Thanks in advance for all the work involved.

[schnaader]

I found what might be a tiny bug [...] Always the issue is related to PDF image streams miscounted.

I opened issue #27 for this, thanks for spotting it!

2) Hundreds of MP3 streams found in a single valid file which aren't processed either.

Known, issue #19, I'll might have a look right after 0.4.5, but it's low priority because it slows down Precomp a bit on such files and distorts MP3 statistics but nothing else.

1) Valid MP3s aren't recognized at all.

Here, an example file would be interesting. Some MP3 types aren't supported by packMP3, so this could be the case, but it also might be some detection/processing bug. Thanks in advance

Trying to speed up the things, I've been using ppx2 to make the batch multi-threaded. But different instances of precomp tries to write their temp files with the same exact name, leading to errors. So, in the meantime precomp still needing those temp files, you might make them unique by giving them a variable as a name, like a random number or <originalfile_########.dat>

I still can't reproduce this, sounds like some sort of race condition because filenames are supposed to be unique already. Anyway, not using temp files anymore is the next thing I'll implement, so I think this is not really worth looking after (I already have too little time for Precomp), sorry.

[Gonzalo]

Known, issue #19, I'll might have a look right after 0.4.5, but it's low priority because it slows down Precomp a bit on such files and distorts MP3 statistics but nothing else.

On the contrary, the slowdown is remarkable. Really, it take ages to give up the file...

New size: 5297324 instead of 5405364

Done.
Time: 50 minute(s), 31 second(s)

5 mb in 50 min... exactly 1.74 kbps = 157x slower than normal on this file. Even worse on others.

Here, an example file would be interesting. Some MP3 types aren't supported by packMP3, so this could be the case, but it also might be some detection/processing bug. Thanks in advance

See attached.

I still can't reproduce this, sounds like some sort of race condition because filenames are supposed to be unique already. Anyway, not using temp files anymore is the next thing I'll implement, so I think this is not really worth looking after (I already have too little time for Precomp), sorry.

It's ok. You better focus on get rid of the temp files for good

[schnaader]

See attached.

This file is MPEG-2 Layer III, one of the formats unsupported by packMP3. I opened issue #28 to be more clear on that in verbose mode. PackMP3 output is better here:

files with errors:
------------------
TEST.mp3 (file is MPEG-2 LAYER III, not supported)

On the contrary, the slowdown is remarkable. Really, it take ages to give up the file...

New size: 5297324 instead of 5405364

Done.
Time: 50 minute(s), 31 second(s)

5 mb in 50 min... exactly 1.74 kbps = 157x slower than normal on this file. Even worse on others.

This is very interesting. Didn't stumble upon such files so far, such an extreme slowdown should be avoided. Could you post a file?

[Gonzalo]

Could you post a file?

[schnaader]

Thanks, opened issue #29 for this and assigned it to the v0.4.5 milestone - I don't want to release the next version without having some kind of fix for this.

[schnaader]

Fixed. The packMP3 error message says "synching failure", which means there's too much garbage data (more than 64K) after the last valid frame. Luckily, it also tells us how long the valid data is so we can simply truncate the MP3 data to this length and retry.

(0.04%) Possible MP3 found at position 2197, length 5403167
Too much garbage data at the end, retry with new length 398106
Best match: 398106 bytes, recompressed to 334933 bytes
(7.41%) Possible MP3 found at position 400303, length 5005061
Too much garbage data at the end, retry with new length 1481143
Best match: 1481143 bytes, recompressed to 1270472 bytes
(34.81%) Possible MP3 found at position 1881446, length 3523918
Too much garbage data at the end, retry with new length 1563167
Best match: 1563167 bytes, recompressed to 1337363 bytes
(63.73%) Possible MP3 found at position 3444613, length 1960751
Too much garbage data at the end, retry with new length 355266
Best match: 355266 bytes, recompressed to 316578 bytes
(70.30%) Possible MP3 found at position 3799879, length 1605485
Too much garbage data at the end, retry with new length 572604
Best match: 572604 bytes, recompressed to 493659 bytes
(80.89%) Possible MP3 found at position 4372483, length 1032881
Too much garbage data at the end, retry with new length 217861
Best match: 217861 bytes, recompressed to 189701 bytes
(84.92%) Possible MP3 found at position 4590344, length 815020
Best match: 815020 bytes, recompressed to 708294 bytes
New size: 4653307 instead of 5405364

Done.
Time: 5 second(s), 242 millisecond(s)

Recompressed streams: 7/7
MP3 streams: 7/7

Recompression is OK, too, I also found and fixed two nasty bugs introduced in the refactoring with the commit. Your file can't be processed by packMP3 (stops with "synching failure" error), but now Precomp can handle it

[Gonzalo]

I can't get it to run on Windows. I use Code::Blocks 16.01, importing .sln
It calls for 2 dlls which i downloaded and copied but still no output.
The dlls are libstdc++-6 and libgcc_s_dw2-1


If anyone is so kind and wants to make an executable.... I have Seven 32 bits.

Edit: Quick test on linux. All my problematic files are processed ok now.
But I'm wondering: Is all that data just "garbage"? The files seemed ok... No player throw error at all, you can just listen they without problem. Could it be a problem of packMP3?

I'll be digging deeper in the next days if I can compile on Windows.

[schnaader]

I can't get it to run on Windows. I use Code::Blocks 16.01, importing .sln
It calls for 2 dlls which i downloaded and copied but still no output.
The dlls are libstdc++-6 and libgcc_s_dw2-1

Sounds like you're using MinGW GCC as compiler, try to add the options "-static -static-libgcc -static-libstdc++", this will remove the need for the DLLs.

If anyone is so kind and wants to make an executable.... I have Seven 32 bits.

Try the attached file. It's the latest version (commit 93c1988).

But I'm wondering: Is all that data just "garbage"? The files seemed ok... No player throw error at all, you can just listen they without problem. Could it be a problem of packMP3?

I wondered that too and will ask Matthias Stirner about it. My guess is that the MP3s are perfectly valid, but some unusual change happens in the MP3 frame format every now and then that packMP3 can't process correctly at the moment, so the file has to be divided into parts where the frame format is the same. Also note that your example file is divided into parts without any gaps (position of first part = 2197, corrected length of first part = 398106, 2197 + 398106 = 400303 = position of second part, and so on), so there even aren't any invalid things or garbage between parts. If my guess is right, this might get changed in packMP3 so it can process this files, too.