Results 1 to 20 of 20

Thread: Paq8px building instructions

  1. #1
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    399
    Thanks
    278
    Thanked 282 Times in 149 Posts

    Paq8px building instructions

    Here is a guide to make a successful paq8px build from source...

    • ...on Windows with Visual Studio Community 2017 and/or MinGW-w64.
    • ...on Linux with GCC and/or Clang + LLVM.

    Extract the attached content, and open howto/index.html.

    Extra:
    If you have Windows and would like to have a Linux development/test platform as well, here you will find a chapter with instrunctions to set up a Virtual Machine with Linux (Ubuntu) equipped with the necessary development tools and shared folders.
    This chapter of the guide is a general one - it may be useful for anyone wishing to do some cross-platform C++ development.

    If you find any typos, errors or have any suggestions regarding the settings, the text, the layout or anything - let me know. I will update this first post.
    Attached Files Attached Files
    Last edited by Gotty; 27th May 2018 at 10:44. Reason: Update

  2. Thanks (10):

    Bulat Ziganshin (30th December 2017),Darek (30th December 2017),Gonzalo (6th June 2018),hexagone (30th December 2017),load (13th January 2018),Mauro Vezzosi (8th April 2018),pinguin2 (28th January 2018),samsat1024 (9th June 2018),schnaader (30th December 2017),Simorq (15th January 2018)

  3. #2
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    241
    Thanks
    41
    Thanked 98 Times in 51 Posts
    Quote Originally Posted by Gotty View Post
    • Linux (currently paq8px does not compile on Linux)


    If you find any typos, errors or have any suggestions
    I suggest making sure it runs correctly on Linux, after you make it compile on Linux. If it produces different compressed files on Linux and Windows, this is a serious issue and should be fixed. I guess cmix has this problem (but I can't run cmix on my computers: it needs much more RAM than they have)

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

  4. #3
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    523
    Thanks
    198
    Thanked 750 Times in 304 Posts
    @Gotty
    v127 doesn't compile on MSVC++ because of the new english stemmer, I'll fix it in the next release.

    @Alexander Rhatushnyak
    paq8px can use itself to pretrain the new exe model, so using that option will always produce different compressed files, even on Windows because each build is different.
    Since paq8px and cmix are just research projects, i.e., not intended for real world usage, I don't think that is a problem.

  5. #4
    Member Alexander Rhatushnyak's Avatar
    Join Date
    Oct 2007
    Location
    Canada
    Posts
    241
    Thanks
    41
    Thanked 98 Times in 51 Posts
    I started compressing LPCB images on 64-bit Ubuntu using Wine and the provided Windows executable of paq8px_v127, this works seemingly well, and hopefully compression will complete tomorrow, and decompression on Jan.2.

    Quote Originally Posted by mpais View Post
    so using that option will always produce different compressed files, even on Windows because each build is different
    If even without that option some compressed files on Linux and Windows are incompatible,
    this should be at least mentioned in the readme.txt, don't you think so?

    This newsgroup is dedicated to image compression:
    http://linkedin.com/groups/Image-Compression-3363256

  6. #5
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    523
    Thanks
    198
    Thanked 750 Times in 304 Posts
    Sure, I guess we can make a simple readme.txt, especially now that more people are getting interested in it.
    I had planned on making documentation explaining not only how to use paq8px, but also its various components and the design decisions that went into making them (the introduction text that you see in the code is very outdated (circa v75, I believe) and a lot has changed), but I don't really have much time for that.
    I've just been focusing on making improvements to it because those can then be ported to paq8pxd and cmix, so it seems to be the best use of my time.

  7. #6
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    581
    Thanks
    225
    Thanked 218 Times in 103 Posts
    Quote Originally Posted by Alexander Rhatushnyak View Post
    I suggest making sure it runs correctly on Linux, after you make it compile on Linux. If it produces different compressed files on Linux and Windows, this is a serious issue and should be fixed. I guess cmix has this problem (but I can't run cmix on my computers: it needs much more RAM than they have)
    Quote Originally Posted by mpais View Post
    @Alexander Rhatushnyak
    paq8px can use itself to pretrain the new exe model, so using that option will always produce different compressed files, even on Windows because each build is different.
    Since paq8px and cmix are just research projects, i.e., not intended for real world usage, I don't think that is a problem.
    As soon as building instructions are available for Linux, I'll create a Dockerfile we can add to the GitHub repository and to Docker Hub (for an example, look at this repository and this Docker hub entry for FLIF). Wanted to do this anyway, but what you describe (being able to reproduce compressed files) even gives a perfect use case now.

    Since Docker virtualizises the whole environment, we get completely reproducable executables that can operate on files outside the Docker container on every platform that supports Docker (64-bit Windows 7 or higher, Linux, OS X). It also provides an alternative to the build instructions for all those platforms: install docker, then 1) run the image directly for an old version or 2) build a new image from the Dockerfile for the latest git version. If needed, we could even provide 3) check out a git version by tag or commit id and build this one.
    http://schnaader.info
    Damn kids. They're all alike.

  8. #7
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    399
    Thanks
    278
    Thanked 282 Times in 149 Posts
    Updated first post with:
    • Windows: How to set up the editor in Visual Studio 2017 to prevent the annoying autoformatting
    • Windows: How to make an optimal MinGW-w64 build

  9. #8
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    399
    Thanks
    278
    Thanked 282 Times in 149 Posts
    Updated first post with:

    • Suggested MinGW-w64 compiler version is changed from [Threads=posix] to [Threads=win32] saving ~23 KB in compiled executable
    • Modified MinGW-w64 build scripts (suppressed file not found errors on cleanup)
    • Supplied a quick build script (without PGO) for MinGW-w64 (see paq8px\mingw\_0_quick_build.cmd)
    • Updated MinGW-w64 build instructions

  10. Thanks (2):

    pinguin2 (28th January 2018),Simorq (15th January 2018)

  11. #9
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    264
    Thanks
    110
    Thanked 153 Times in 112 Posts
    I followed Paq8px building instructions to build a quick (_0_quick_build) and a profiled (_1_build_gen_profile + _2_run_paq + _3_build_use_profile) versions with g++ (x86_64-win32-sjlj-rev1, Built by MinGW-W64 project) 7.1.0.

    2 Maximum Compression files has different compressed size:
    Code:
    Quick build:
      857.915 AcroRd32.exe.paq8px141
    1.214.818 MSO97.DLL.paq8px141
    
    Profiled build:
      857.948 AcroRd32.exe.paq8px141
    1.214.793 MSO97.DLL.paq8px141
    "Building paq8px from source": in paq8px\howto\index2_zlib_source.html the link "MinGW-w64 setup" at the bottom of the page is broken, it should links to index41_mingw_install.html, not to paq8px\howto\index4_mingw.html.

    Running _2_run_paq.cmd gives "Specified input is a directory but should be a file: ..\profile-suite" and paq8px exits immediately.

  12. #10
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    155
    Thanks
    15
    Thanked 17 Times in 15 Posts
    Quote Originally Posted by Mauro Vezzosi View Post
    I followed Paq8px building instructions to build a quick (_0_quick_build) and a profiled (_1_build_gen_profile + _2_run_paq + _3_build_use_profile) versions with g++ (x86_64-win32-sjlj-rev1, Built by MinGW-W64 project) 7.1.0.

    2 Maximum Compression files has different compressed size:
    Code:
    Quick build:
      857.915 AcroRd32.exe.paq8px141
    1.214.818 MSO97.DLL.paq8px141
    
    Profiled build:
      857.948 AcroRd32.exe.paq8px141
    1.214.793 MSO97.DLL.paq8px141
    "Building paq8px from source": in paq8px\howto\index2_zlib_source.html the link "MinGW-w64 setup" at the bottom of the page is broken, it should links to index41_mingw_install.html, not to paq8px\howto\index4_mingw.html.

    Running _2_run_paq.cmd gives "Specified input is a directory but should be a file: ..\profile-suite" and paq8px exits immediately.
    Is prediction or probability counting in paq not stable ?

  13. #11
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    399
    Thanks
    278
    Thanked 282 Times in 149 Posts
    It is stable - as much as I have tested it so far. The result must be the same when compiling it with a different compiler (Visual C++ in my case) or with the same compiler with different optimization options (MinGW-w64). I'll look into it.
    I suspect a processor bug: it probably kicks in when the system (data registers, flags, memory) is in a specific state.

    @Mauro, did you encounter the problem on your i3-3240?

  14. #12
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    264
    Thanks
    110
    Thanked 153 Times in 112 Posts
    Quote Originally Posted by Gotty View Post
    @Mauro, did you encounter the problem on your i3-3240?
    I can't test it now because I haven't the quick build on i3-3240, tomorrow I'll run the quick and the profiled builds on it (my previous tests was on i7-4710HQ).

  15. #13
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    399
    Thanks
    278
    Thanked 282 Times in 149 Posts
    Oh, I thought the "difference" problem is on your i3-3240. So you have "difference" problems on both systems?

  16. #14
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    399
    Thanks
    278
    Thanked 282 Times in 149 Posts
    I could reproduce the problem. Thanx for all the info! I'm on it.
    It's not a cpu issue.

    Edit:
    Narrowed it down. The "difference" problem between a quick build and a profiled build is caused by the -ftree-vrp optimization switch present at -O2 and higher.
    Quick solution: if you include -fno-tree-vrp optimization switch the problem goes away.
    The problem is present only with the quick build.
    Something in the source code confuses/bothers GCC. I'll find it.
    Last edited by Gotty; 9th April 2018 at 23:53.

  17. #15
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    264
    Thanks
    110
    Thanked 153 Times in 112 Posts
    > Oh, I thought the "difference" problem is on your i3-3240. So you have "difference" problems on both systems?
    I found it on i7-4710HQ and I haven't tested it on i3-3240.
    I guess I can avoid to test it also on i3-3240, now we know where is the problem.

    > I'll find it.
    It seems to me that it's not easy to find the problem in paq8px.
    Could it be a problem in g++ instead? My version is 7.1.0, now there is 7.3.0.
    Thank you for everything you do!

  18. #16
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    399
    Thanks
    278
    Thanked 282 Times in 149 Posts
    Yes. You don't need more tests.
    The proplem exists starting from v140. I have the exact line of the code now (in FrenchStemmer::Stemmer()). But it does not make any sense, yet. The code looks all right to me.
    It is probably an uninitialized variable somewhere or some "light" memory corruption, or buffer overflow. I already found an uninitialized variable (Word::Language), fixed, but it was not that.
    My search goes on. I will come back soon.

    Edit: fixed in v141fix1. See the paq8px thread.
    Last edited by Gotty; 26th May 2018 at 20:07. Reason: Found the problem

  19. Thanks (2):

    Mauro Vezzosi (10th April 2018),Mike (10th April 2018)

  20. #17
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    399
    Thanks
    278
    Thanked 282 Times in 149 Posts
    Updated first post with:

    • Added instructions for Linux/GCC
    • Added instructions for Linux/Clang+LLVM
    • Modified the Windows/MinGW-w64 build scripts to make a faster executable in accordance with https://encode.su/threads/342-paq8px...ll=1#post55981
    • Reorganized file names.
    • All pngs are now 8bit instead of 24bit + recompressed them to make the guide smaller.
    • Fixed problems reported by Mauro: fixed broken link and updated the pgo instructions for newer paq8px versions with the new command line interface (v137+)

    Extra:
    If you have Windows and would like to have a Linux development/test platform as well, here you will find a chapter with instrunctions to set up a Virtual Machine with Linux (Ubuntu) equipped with the necessary development tools and shared folders.
    This chapter of the guide is a general one - it may be useful for anyone wishing to do some cross-platform C++ development.

  21. Thanks:

    Darek (26th May 2018)

  22. #18
    Member
    Join Date
    Aug 2014
    Location
    Argentina
    Posts
    536
    Thanks
    236
    Thanked 90 Times in 70 Posts
    Hello Gotty! I just downloaded your guide. That is one comprehensive document! I'm impressed

    I have a quick question, though, if you don't mind: How much faster can profiling make the binary in your experience? Some 10-15% would be just great for me. Is it reasonable to expect something like that?

    As a suggestion, if you have this info, it could be included in the guide...

  23. #19
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    399
    Thanks
    278
    Thanked 282 Times in 149 Posts
    Hello Gonzalo! Thanx for the feedback!
    In case of paq8px the difference is just 1-2%. Sometimes they run at the same speed. But the profiled exe is much smaller.
    I have still some things on my to do list: an appendix with the useful gcc optimization switches with speed and size comparison to other compilers; and a "How to make a profiled build with Visual Studio". Maybe around Christmas.
    Last edited by Gotty; 7th June 2018 at 01:50.

  24. Thanks:

    Gonzalo (8th June 2018)

  25. #20
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Hungary
    Posts
    399
    Thanks
    278
    Thanked 282 Times in 149 Posts
    Warning:
    Avoid using -ffast-math (/fp:fast in vc++) when compiling paq8px.

    More: https://encode.su/threads/342-paq8px...ll=1#post58227

Similar Threads

  1. paq8px
    By Jan Ondrus in forum Data Compression
    Replies: 1820
    Last Post: Today, 09:38
  2. benchmark of mixed data types with SIMD instructions
    By just a worm in forum The Off-Topic Lounge
    Replies: 8
    Last Post: 10th June 2015, 23:28
  3. PAQ8PX for Mac OS X?
    By Shoal in forum Data Compression
    Replies: 11
    Last Post: 8th July 2011, 18:35
  4. Alternative paq8px builds
    By M4ST3R in forum Download Area
    Replies: 20
    Last Post: 25th June 2010, 17:19
  5. Replies: 0
    Last Post: 26th July 2007, 19:47

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •