Page 2 of 15 FirstFirst 123412 ... LastLast
Results 31 to 60 of 450

Thread: cmix

  1. #31
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    243
    Thanks
    112
    Thanked 114 Times in 69 Posts
    I have a new fix for cmix v6 Windows version. I think this should improve it to match the performance of the Linux version. Unfortunately I am using "Windows 7 Home Premium" which has a 16GB RAM limit, so I can't test to see if it works. Could someone test it out on enwik8 to confirm?

    The fix is in paq8hp.cpp (changing one line of code: "U8 *p=&t[i*B-B];" to "U8 *p=&t[i*B]-B;").
    Attached Files Attached Files

  2. Thanks:

    Skymmer (8th December 2014)

  3. #32
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,030
    Thanks
    625
    Thanked 418 Times in 316 Posts
    I can't run it, because there is request for cygwin1.dll - I had installed this librarry and it still doesn't work....
    Last edited by Darek; 8th December 2014 at 12:54.

  4. #33
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    688
    Thanks
    41
    Thanked 173 Times in 88 Posts
    Thats a good news. Thanks Byron!
    I'll will create win compile as soon I'll return from work. It will be at 8pm (GMT+3).

    Darek, the compile provided by Byron created using specific mmmm... lets call it branch of GCC. More exactly speaking with 4.8.2 20131016 (Fedora Cygwin). I suppose some specific cygwin may required.
    Also I'm afraid that cross-compiled cygwin based version of cmix may perform quite slowly
    Last edited by Skymmer; 8th December 2014 at 16:57.

  5. #34
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,030
    Thanks
    625
    Thanked 418 Times in 316 Posts
    Ok, then what exacltly I suppose to do to run it properly? Actually despite cygwin installed this version call that cygwin1 is not present.
    Do I need to install This version of Fedora Cygwin?

  6. #35
    Member
    Join Date
    May 2009
    Location
    France
    Posts
    99
    Thanks
    13
    Thanked 75 Times in 45 Posts
    Hello,

    cmix_v6_win64.7z, NOT tested.

    AiZ
    Last edited by AiZ; 8th December 2014 at 22:20.

  7. #36
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    688
    Thanks
    41
    Thanked 173 Times in 88 Posts
    Bad news. I was not able to create the compile which gives the same size on enwik6 as stated at cmix site (http://www.byronknoll.com/cmix.html)
    The size given there is 181908 bytes. I got 181915 or 181917 or even 181929. Seems compiler optimization options have effect on resulting size.
    Currently I don't know how to achieve the same one.

    P.S. My results are with -c english.dic enwik6 out.cmix

    AiZ, your compile crashes on my system.
    Last edited by Skymmer; 8th December 2014 at 22:54.

  8. #37
    Member
    Join Date
    May 2009
    Location
    France
    Posts
    99
    Thanks
    13
    Thanked 75 Times in 45 Posts
    Ok,
    Quote Originally Posted by Skymmer View Post
    AiZ, your compile crashes on my system.
    I'll test it tomorrow at work, I've got a handful of Win2K8 workstations with 108GB of RAM available.

    AiZ

  9. #38
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    243
    Thanks
    112
    Thanked 114 Times in 69 Posts
    Quote Originally Posted by Skymmer View Post
    Bad news. I was not able to create the compile which gives the same size on enwik6 as stated at cmix site (http://www.byronknoll.com/cmix.html)
    The size given there is 181908 bytes. I got 181915 or 181917 or even 181929. Seems compiler optimization options have effect on resulting size.
    Currently I don't know how to achieve the same one.

    P.S. My results are with -c english.dic enwik6 out.cmix

    AiZ, your compile crashes on my system.
    Ah, I wasn't expecting it to get exactly the same output as the linux version - as you said the compiler optimization options can change the output. However, it should perform approximately as well. Skymmer, could you post your cmix.exe compile, since the one I posted earlier doesn't seem to work for other people? I would be curious to see how it does on enwik8 (with a decompression test as well to verify it successfully reproduces the original file).

  10. #39
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    688
    Thanks
    41
    Thanked 173 Times in 88 Posts
    Quote Originally Posted by byronknoll View Post
    Skymmer, could you post your cmix.exe compile
    Sure, here it is.
    Attached Files Attached Files

  11. #40
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,030
    Thanks
    625
    Thanked 418 Times in 316 Posts
    Now it works. Thank you Byron Skymmer!

    I'll test enwik8.

    After first short tests - this version is a little bit better than your latest compile from 4th of November - in average by 0,004% - differences are in bytes. Some files are comresssed worse, most - better.

  12. #41
    Member
    Join Date
    May 2009
    Location
    France
    Posts
    99
    Thanks
    13
    Thanked 75 Times in 45 Posts
    Hello,

    FYI, with Skymmer version
    Code:
    D:\Tmp\cmix>timer64 cmix -c english.dic enwik8 enwik8.paq
    
     cmix-v6-win64 build: 20141208 21:23
     Number of models:      1728
     Num. of mixer neurons: 414264
    100000000 bytes -> 15776674 bytes in 45208.29 s.
    cross entropy: 1.262
    
    
    Kernel  Time =    21.044 =    0%
    User    Time = 45199.792 =   99%
    Process Time = 45220.836 =   99%    Virtual  Memory =  28427 MB
    Global  Time = 45225.501 =  100%    Physical Memory =  18236 MB
    and, when compiled with 64bit g++.exe (Rev2, Built by MSYS2 project) 4.9.2
    Code:
    Program received signal SIGSEGV, Segmentation fault.
    0x000000000041da0d in (anonymous namespace)::BH<4>::operator[] (
        this=0xe2a1a0 <_ZZN12_GLOBAL__N_113contextModel2EvE4rcm7>, i=0)
        at src/models/paq8hp.cpp:689
    689         if (p[2]==0) { *(U16*)p=chk; break; }
    AiZ

  13. #42
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    243
    Thanks
    112
    Thanked 114 Times in 69 Posts
    Thanks Skymmer/AiZ/Darek for the testing. It looks like my fix was unsuccessful - the Windows compile should be getting 15,73x,xxx bytes on enwik8. It is hard to debug the issue when I can't test on my own computer! I will keep working on it.

  14. #43
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,030
    Thanks
    625
    Thanked 418 Times in 316 Posts
    Enwik8 test with dictionary option:

    compressed size: 15'776'674 bytes = 0,01% better than previous compile.
    compression time: 31'140s = 10% faster than previous compile.
    decompression time: 30'236s. Checksum verified, ok.

    That exact as AiZ scores.

    Darek

    p.s. hmmmm, one question - which dictionary will you use to test? I've used cmix6 dictionary. Maybe some differences in this file could provide to different scores.
    Last edited by Darek; 12th December 2014 at 09:23.

  15. #44
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    243
    Thanks
    112
    Thanked 114 Times in 69 Posts
    Quote Originally Posted by Darek View Post
    p.s. hmmmm, one question - which dictionary will you use to test? I've used cmix6 dictionary. Maybe some differences in this file could provide to different scores.
    Yeah you should use the cmix6 dictionary (that is what I use for all benchmarks).

  16. #45
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    243
    Thanks
    112
    Thanked 114 Times in 69 Posts
    Good news. I upgraded to "Windows 7 Ultimate", so now I have enough available RAM to test cmix. I tried testing my compile above on enwik8:

    Code:
    $./cmix -c dictionary/english.dic data/enwik8 data/out.dat
    100000000 bytes -> 15738924 bytes in 30872.32 s.
    cross entropy: 1.259
    Two bytes off from the Linux version (which was 15738922). So it looks like compiling (with my fix to paq8hp.cpp) in cygwin with g++ works.

  17. #46
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,030
    Thanks
    625
    Thanked 418 Times in 316 Posts
    Byron: Then we could find how upgrade system to yours cygwin library.

    Ok, I've installed cygwin from this link: http://www.cygwinports.com/

    and it works, however for dictionary option for most files I've got error:

    0 [main] cmix6 5816 cygwin_exception:pen_stackdumpfile: Dumping stack trace to cmix6.exe.stackdump

    Maybe it's physical memory problem - I've got only 16GB of ram. However Skymmer compiles doesn't crush and works with swap file correctly.

    I've tested this compile on my testbed files and compare to previous builds (all in nondictionary mode) and in overall it's compression ratio is about 0,014% worse than Skymmer compile from 9.12.2014. Similar decrease of compression ratio I've seen on few files which run with dictionary option.

    Maybe big text files with dictionary option are compress better. I can't test enwik8 file...

    Darek
    Last edited by Darek; 14th December 2014 at 12:31.

  18. Thanks:

    Mangix (12th December 2014)

  19. #47
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,257
    Thanks
    307
    Thanked 795 Times in 488 Posts
    Brian Knoll released cmix v8. http://www.byronknoll.com/cmix.html
    It already had the number 1 spot on both LTCB and Silesia. The new version improves compression even further.
    However, it takes 4 days to compress or decompress enwik9 on a 32 GB i7-3770.
    His test results are as follows:

    Version: 8
    Compressed size of enwik8: 15709216 bytes
    Compressed size of enwik9: 123930173 bytes
    The program doesn't contain any configurable options.

    Size of source code as a zip file: 164,882 bytes
    zip file contains:
    - all source code
    - makefile
    - dictionary (compressed using cmix from 465211 bytes -> 89713 bytes)

    enwik9 compression time: 344244.32 seconds
    enwik9 decompression time: 346641.12 seconds
    Approximate memory used: 30311032 KiB

    Description of test machine:
    processor: Intel Core i7-3770K
    memory: 31.4 GiB
    OS: Linux Mint 17

    Description and source code: http://www.byronknoll.com/cmix.html

    Silesia compressed size in bytes:

    dicke: 1893344
    mozil: 9565086
    mr: 2020572
    nci: 838631
    ooff: 1338376
    osdb: 2020534
    reym: 770535
    samba: 2591823
    sao: 3767137
    webst: 4689079
    x-ray: 3568206
    xml: 244270

    total: 33307593

  20. Thanks (4):

    comp1 (11th November 2015),Cyan (11th November 2015),Hobart (12th November 2015),Skymmer (12th November 2015)

  21. #48
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,030
    Thanks
    625
    Thanked 418 Times in 316 Posts
    Quote Originally Posted by Matt Mahoney View Post
    Brian Knoll released cmix v8. http://www.byronknoll.com/cmix.html
    Matt,
    there are small typo in samba score on Silesia test scores page - now the score is 2951 instead 2591.

    Question/request - if anyone could made Win64 build of this version of CMIX?

    Darek

  22. Thanks:

    Matt Mahoney (12th November 2015)

  23. #49
    Member przemoc's Avatar
    Join Date
    Aug 2011
    Location
    Poland
    Posts
    44
    Thanks
    3
    Thanked 23 Times in 13 Posts
    Quote Originally Posted by Darek View Post
    Question/request - if anyone could made Win64 build of this version of CMIX?
    I've built it in MSYS2's MinGW-w64 Win64 Shell using recent gcc version 5.2.0 (Rev4, Built by MSYS2 project) with
    Code:
    all: LFLAGS += -static -static-libgcc -static-libstdc++
    added to Makefile to include non-win dependencies. The thing is I cannot test whether this build is good, because my notebook quickly runs out of memory...
    Attached Files Attached Files

  24. #50
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,030
    Thanks
    625
    Thanked 418 Times in 316 Posts
    At first very thanks for this compile.

    However it doesn't work in my computer.

    Quote Originally Posted by przemoc View Post
    I've built it in MSYS2's MinGW-w64 Win64 Shell using recent gcc version 5.2.0 (Rev4, Built by MSYS2 project)
    I've already installed older version of MiniGW and probably due to this program crashes and calls:

    "
    terminate called after throwing an instance of 'char const*'
    This application has requested the Runtime to terminate it in an unusual way. Please contact the application's suport team for more information.
    "

    Where I could find and download proper version of MiniGW-w64?

    Darek

  25. #51
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    243
    Thanks
    112
    Thanked 114 Times in 69 Posts
    I have tried compiling cmix for Windows using both cygwin and mingw. Compiling with cygwin should work without any problems. However, it is not convenient to widely distribute that binary since it is incompatible with other versions of cygwin. I haven't been able to get cmix successfully compiled using mingw. Looking at older messages in this thread I see Skymmer compiled a version that works: "cmix-v6-win64-fixed-paq8hp-model.7z". I wonder how Skymmer compiled it. It would be great if I could include an official Windows binary along with the cmix releases.

  26. #52
    Member
    Join Date
    Apr 2009
    Location
    here
    Posts
    204
    Thanks
    171
    Thanked 110 Times in 66 Posts
    i compiled it too for windows, but it crashes as well. maybe not enough memory?

    This application has requested the Runtime to terminate it in an unusual way.
    Please contact the application's support team for more information.
    terminate called after throwing an instance of 'std::bad_alloc'
    what(): std::bad_alloc
    /edit : attached another version, with only -O3 instead of -Ofast, still nock luck here.

    i will try on my desktop pc later, with 32gb ram.
    Attached Files Attached Files
    Last edited by load; 13th November 2015 at 07:59.

  27. #53
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,030
    Thanks
    625
    Thanked 418 Times in 316 Posts
    Quote Originally Posted by byronknoll View Post
    I Looking at older messages in this thread I see Skymmer compiled a version that works: "cmix-v6-win64-fixed-paq8hp-model.7z". I wonder how Skymmer compiled it.
    For my computer setup Skymmer version was the best and the most stable at all. It won't crashes nor hangs. Other versions have problems with dictionary option for many files...

    Maybe it's worth to mention that I have two laptops - with 16GB and 32GB of memory, however this latest compile (as well as not Skymmer's builds) crashes on both machines. From other side Skymmer's version works fine on both machines as well - even on 16GB computer. Of course 16GB machine was much slower due to swap memory usage, but I've tested all files w/o any issues on this laptop too.

    Then in my opinion it's not directly memory problem - maybe some issues with memory allocations?

    Darek

    @load: - I've the same error as you with your compile but w/o part of message about memory allocation:"terminate called after throwing an instance of 'std::bad_alloc'
    what(): std::bad_alloc"

    Last edited by Darek; 13th November 2015 at 11:15.

  28. #54
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,030
    Thanks
    625
    Thanked 418 Times in 316 Posts
    Request for Skymmer: could you try to compile your build of the latest version od CMIX (Ver.8 )?

    Darek

  29. #55
    Member przemoc's Avatar
    Join Date
    Aug 2011
    Location
    Poland
    Posts
    44
    Thanks
    3
    Thanked 23 Times in 13 Posts
    Quote Originally Posted by Darek View Post
    At first very thanks for this compile.

    However it doesn't work in my computer.
    Well, maybe this build is dysfunctional. As I wrote earlier, I quickly get:
    Code:
    terminate called after throwing an instance of 'std::bad_alloc'
      what():  std::bad_alloc
    
    This application has requested the Runtime to terminate it in an unusual way.
    Please contact the application's support team for more information.
    when trying to compress something.

    Quote Originally Posted by Darek View Post
    I've already installed older version of MiniGW and probably due to this program crashes and calls:

    "
    terminate called after throwing an instance of 'char const*'
    This application has requested the Runtime to terminate it in an unusual way. Please contact the application's suport team for more information.
    "
    Your already installed MinGW shouldn't matter here, because I built the exe statically (by modifying Makefile as I wrote earlier).

    Quote Originally Posted by Darek View Post
    Where I could find and download proper version of MiniGW-w64?
    MinGW-w64 has many different builds. Bare compiler is not that convenient to use if you want to build POSIX-y applications, requiring some basic UNIX utilities (like make, awk, sed, grep, rm, etc.).

    If you want native windows build with basic stuff ready (or easily ready) to use, then MSYS2 is nicely wrapped bootstrapping environment, where you can easily download most stuff already prebuilt.
    Another nice MinGW-w64 distro is done by STL. Nowadays it's MSYS2-based too (EDIT: or at least it's built using MSYS2 according to its page, haven't used it myself for a few years). It has even git within it, but there is git-less installer too.

    For cross-compiling from Linux in the past I was using Ruben's builds (he no longer provides new versions) and Win-builds (previously known as yypkg mingw-builds). Nowadays many popular distributions have mingw-w64 in their repositories.
    Last edited by przemoc; 21st November 2015 at 22:20. Reason: addendum

  30. Thanks:

    Darek (23rd November 2015)

  31. #56
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    688
    Thanks
    41
    Thanked 173 Times in 88 Posts
    Well, I'm trying to get the working binary all this week and I'm almost ready to give up. God damned SIGSEGV is back again. Debugger says:
    Program received signal SIGSEGV, Segmentation fault.
    0x000000007701f280 in ntdll!RtlIsValidIndexHandle () from C:\Windows\SYSTEM32\ntdll.dll


    Probably its possible to find out more but I'm simply don't have any debugging skills to make it right.
    I have one theory why we have problems compiling cmix on Windows but its only an idea.
    Shortly: LP64 data model vs. LLP64 one. More info by the links below:
    https://cygwin.com/cygwin-ug-net/pro...ng.html#gcc-64
    http://stackoverflow.com/questions/7...ng-in-64-bit-c
    https://en.wikipedia.org/wiki/64-bit...it_data_models

    Anyway its nice to see that cmix development goes on.

  32. #57
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    243
    Thanks
    112
    Thanked 114 Times in 69 Posts
    Hi Skymmer, can you give more information on how you compiled "cmix-v6-win64-fixed-paq8hp-model.7z"? Which compiler did you use? Thanks!

  33. #58
    Member Skymmer's Avatar
    Join Date
    Mar 2009
    Location
    Russia
    Posts
    688
    Thanks
    41
    Thanked 173 Times in 88 Posts
    Actually its based on fix provided by you in post#31 in this thread. Plus some modifications.

    predictor.cpp
    added #include <stdio.h>

    runner.cpp
    added #include <cstdlib>

    paq8hp.cpp
    changed U8 *p=&t[i*B-B]; to U8 *p=&t[i*B] - B;
    changed throw message; to printf("%s",message);

    paq8l.cpp
    line 56: changed #ifndef WINDOWS to #ifndef min
    changed throw message; to printf("%s",message);

    Compiler: Nuwen 12.2 MinGW Distro (GCC 4.9.2 based)
    nuwen.net
    Last edited by Skymmer; 22nd November 2015 at 03:29.

  34. Thanks (3):

    Bulat Ziganshin (22nd November 2015),byronknoll (22nd November 2015),Darek (23rd November 2015)

  35. #59
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,030
    Thanks
    625
    Thanked 418 Times in 316 Posts
    @Przemoc - thank you for full description. I'll try to run this biuld on other computers. Maybe it will works...

    @Skymmer - also thank you for effort to made biuld. As it looks all Cmix versions were hard to compile and made proper working exec on Windows.

  36. #60
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,030
    Thanks
    625
    Thanked 418 Times in 316 Posts
    @Skymmer - have you now stable build of cmix8 for win64?

Page 2 of 15 FirstFirst 123412 ... LastLast

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •