Paq8px building instructions

[Gotty]

Here is a guide to make a successful paq8px build from source...

• ...on Windows with Visual Studio Community 2017 and/or MinGW-w64.
• ...on Linux with GCC and/or Clang + LLVM.

Extract the attached content, and open howto/index.html.

Extra:
If you have Windows and would like to have a Linux development/test platform as well, here you will find a chapter with instrunctions to set up a Virtual Machine with Linux (Ubuntu) equipped with the necessary development tools and shared folders.
This chapter of the guide is a general one - it may be useful for anyone wishing to do some cross-platform C++ development.

If you find any typos, errors or have any suggestions regarding the settings, the text, the layout or anything - let me know. I will update this first post.

[Alexander Rhatushnyak]

• Linux (currently paq8px does not compile on Linux)

If you find any typos, errors or have any suggestions

I suggest making sure it runs correctly on Linux, after you make it compile on Linux. If it produces different compressed files on Linux and Windows, this is a serious issue and should be fixed. I guess cmix has this problem (but I can't run cmix on my computers: it needs much more RAM than they have)

[mpais]

@Gotty
v127 doesn't compile on MSVC++ because of the new english stemmer, I'll fix it in the next release.

@Alexander Rhatushnyak
paq8px can use itself to pretrain the new exe model, so using that option will always produce different compressed files, even on Windows because each build is different.
Since paq8px and cmix are just research projects, i.e., not intended for real world usage, I don't think that is a problem.

[Alexander Rhatushnyak]

I started compressing LPCB images on 64-bit Ubuntu using Wine and the provided Windows executable of paq8px_v127, this works seemingly well, and hopefully compression will complete tomorrow, and decompression on Jan.2.

so using that option will always produce different compressed files, even on Windows because each build is different

If even without that option some compressed files on Linux and Windows are incompatible,
this should be at least mentioned in the readme.txt, don't you think so?

[mpais]

Sure, I guess we can make a simple readme.txt, especially now that more people are getting interested in it.
I had planned on making documentation explaining not only how to use paq8px, but also its various components and the design decisions that went into making them (the introduction text that you see in the code is very outdated (circa v75, I believe) and a lot has changed), but I don't really have much time for that.
I've just been focusing on making improvements to it because those can then be ported to paq8pxd and cmix, so it seems to be the best use of my time.

[schnaader]

I suggest making sure it runs correctly on Linux, after you make it compile on Linux. If it produces different compressed files on Linux and Windows, this is a serious issue and should be fixed. I guess cmix has this problem (but I can't run cmix on my computers: it needs much more RAM than they have)

@Alexander Rhatushnyak
paq8px can use itself to pretrain the new exe model, so using that option will always produce different compressed files, even on Windows because each build is different.
Since paq8px and cmix are just research projects, i.e., not intended for real world usage, I don't think that is a problem.

As soon as building instructions are available for Linux, I'll create a Dockerfile we can add to the GitHub repository and to Docker Hub (for an example, look at this repository and this Docker hub entry for FLIF). Wanted to do this anyway, but what you describe (being able to reproduce compressed files) even gives a perfect use case now.

Since Docker virtualizises the whole environment, we get completely reproducable executables that can operate on files outside the Docker container on every platform that supports Docker (64-bit Windows 7 or higher, Linux, OS X). It also provides an alternative to the build instructions for all those platforms: install docker, then 1) run the image directly for an old version or 2) build a new image from the Dockerfile for the latest git version. If needed, we could even provide 3) check out a git version by tag or commit id and build this one.

[Gotty]

Updated first post with:

• Windows: How to set up the editor in Visual Studio 2017 to prevent the annoying autoformatting
• Windows: How to make an optimal MinGW-w64 build

[Gotty]

Updated first post with:

• Suggested MinGW-w64 compiler version is changed from [Threads=posix] to [Threads=win32] saving ~23 KB in compiled executable
• Modified MinGW-w64 build scripts (suppressed file not found errors on cleanup)
• Supplied a quick build script (without PGO) for MinGW-w64 (see paq8px\mingw\_0_quick_build.cmd)
• Updated MinGW-w64 build instructions

[Mauro Vezzosi]

I followed Paq8px building instructions to build a quick (_0_quick_build) and a profiled (_1_build_gen_profile + _2_run_paq + _3_build_use_profile) versions with g++ (x86_64-win32-sjlj-rev1, Built by MinGW-W64 project) 7.1.0.

2 Maximum Compression files has different compressed size:

Quick build:
  857.915 AcroRd32.exe.paq8px141
1.214.818 MSO97.DLL.paq8px141

Profiled build:
  857.948 AcroRd32.exe.paq8px141
1.214.793 MSO97.DLL.paq8px141

"Building paq8px from source": in paq8px\howto\index2_zlib_source.html the link "MinGW-w64 setup" at the bottom of the page is broken, it should links to index41_mingw_install.html, not to paq8px\howto\index4_mingw.html.

Running _2_run_paq.cmd gives "Specified input is a directory but should be a file: ..\profile-suite" and paq8px exits immediately.

[suryakandau@yahoo.co.id]

I followed Paq8px building instructions to build a quick (_0_quick_build) and a profiled (_1_build_gen_profile + _2_run_paq + _3_build_use_profile) versions with g++ (x86_64-win32-sjlj-rev1, Built by MinGW-W64 project) 7.1.0.

2 Maximum Compression files has different compressed size:

Quick build:
  857.915 AcroRd32.exe.paq8px141
1.214.818 MSO97.DLL.paq8px141

Profiled build:
  857.948 AcroRd32.exe.paq8px141
1.214.793 MSO97.DLL.paq8px141

"Building paq8px from source": in paq8px\howto\index2_zlib_source.html the link "MinGW-w64 setup" at the bottom of the page is broken, it should links to index41_mingw_install.html, not to paq8px\howto\index4_mingw.html.

Running _2_run_paq.cmd gives "Specified input is a directory but should be a file: ..\profile-suite" and paq8px exits immediately.

Is prediction or probability counting in paq not stable ?

[Gotty]

It is stable - as much as I have tested it so far. The result must be the same when compiling it with a different compiler (Visual C++ in my case) or with the same compiler with different optimization options (MinGW-w64). I'll look into it.
I suspect a processor bug: it probably kicks in when the system (data registers, flags, memory) is in a specific state.

@Mauro, did you encounter the problem on your i3-3240?

[Mauro Vezzosi]

@Mauro, did you encounter the problem on your i3-3240?

I can't test it now because I haven't the quick build on i3-3240, tomorrow I'll run the quick and the profiled builds on it (my previous tests was on i7-4710HQ).

[Gotty]

Oh, I thought the "difference" problem is on your i3-3240. So you have "difference" problems on both systems?

[Gotty]

I could reproduce the problem. Thanx for all the info! I'm on it.
It's not a cpu issue.

Edit:
Narrowed it down. The "difference" problem between a quick build and a profiled build is caused by the -ftree-vrp optimization switch present at -O2 and higher.
Quick solution: if you include -fno-tree-vrp optimization switch the problem goes away.
The problem is present only with the quick build.
Something in the source code confuses/bothers GCC. I'll find it.

[Mauro Vezzosi]

> Oh, I thought the "difference" problem is on your i3-3240. So you have "difference" problems on both systems?
I found it on i7-4710HQ and I haven't tested it on i3-3240.
I guess I can avoid to test it also on i3-3240, now we know where is the problem.

> I'll find it.
It seems to me that it's not easy to find the problem in paq8px.
Could it be a problem in g++ instead? My version is 7.1.0, now there is 7.3.0.
Thank you for everything you do!

[Gotty]

Yes. You don't need more tests.
The proplem exists starting from v140. I have the exact line of the code now (in FrenchStemmer::Stemmer()). But it does not make any sense, yet. The code looks all right to me.
It is probably an uninitialized variable somewhere or some "light" memory corruption, or buffer overflow. I already found an uninitialized variable (Word::Language), fixed, but it was not that.
My search goes on. I will come back soon.

Edit: fixed in v141fix1. See the paq8px thread.

[Gotty]

Updated first post with:

• Added instructions for Linux/GCC
• Added instructions for Linux/Clang+LLVM
• Modified the Windows/MinGW-w64 build scripts to make a faster executable in accordance with https://encode.su/threads/342-paq8px...ll=1#post55981
• Reorganized file names.
• All pngs are now 8bit instead of 24bit + recompressed them to make the guide smaller.
• Fixed problems reported by Mauro: fixed broken link and updated the pgo instructions for newer paq8px versions with the new command line interface (v137+)

Extra:
If you have Windows and would like to have a Linux development/test platform as well, here you will find a chapter with instrunctions to set up a Virtual Machine with Linux (Ubuntu) equipped with the necessary development tools and shared folders.
This chapter of the guide is a general one - it may be useful for anyone wishing to do some cross-platform C++ development.

[Gonzalo]

Hello Gotty! I just downloaded your guide. That is one comprehensive document! I'm impressed

I have a quick question, though, if you don't mind: How much faster can profiling make the binary in your experience? Some 10-15% would be just great for me. Is it reasonable to expect something like that?

As a suggestion, if you have this info, it could be included in the guide...

[Gotty]

Hello Gonzalo! Thanx for the feedback!
In case of paq8px the difference is just 1-2%. Sometimes they run at the same speed. But the profiled exe is much smaller.
I have still some things on my to do list: an appendix with the useful gcc optimization switches with speed and size comparison to other compilers; and a "How to make a profiled build with Visual Studio". Maybe around Christmas.

[Gotty]

Warning:
Avoid using -ffast-math (/fp:fast in vc++) when compiling paq8px.

More: https://encode.su/threads/342-paq8px...ll=1#post58227