cmix

[byronknoll]

I have a new fix for cmix v6 Windows version. I think this should improve it to match the performance of the Linux version. Unfortunately I am using "Windows 7 Home Premium" which has a 16GB RAM limit, so I can't test to see if it works. Could someone test it out on enwik8 to confirm?

The fix is in paq8hp.cpp (changing one line of code: "U8 *p=&t[i*B-B];" to "U8 *p=&t[i*B]-B;").

[Darek]

I can't run it, because there is request for cygwin1.dll - I had installed this librarry and it still doesn't work....

[Skymmer]

Thats a good news. Thanks Byron!
I'll will create win compile as soon I'll return from work. It will be at 8pm (GMT+3).

Darek, the compile provided by Byron created using specific mmmm... lets call it branch of GCC. More exactly speaking with 4.8.2 20131016 (Fedora Cygwin). I suppose some specific cygwin may required.
Also I'm afraid that cross-compiled cygwin based version of cmix may perform quite slowly

[Darek]

Ok, then what exacltly I suppose to do to run it properly? Actually despite cygwin installed this version call that cygwin1 is not present.
Do I need to install This version of Fedora Cygwin?

[AiZ]

Hello,

cmix_v6_win64.7z, NOT tested.

AiZ

[Skymmer]

Bad news. I was not able to create the compile which gives the same size on enwik6 as stated at cmix site (http://www.byronknoll.com/cmix.html)
The size given there is 181908 bytes. I got 181915 or 181917 or even 181929. Seems compiler optimization options have effect on resulting size.
Currently I don't know how to achieve the same one.

P.S. My results are with -c english.dic enwik6 out.cmix

AiZ, your compile crashes on my system.

[AiZ]

Ok,

AiZ, your compile crashes on my system.

I'll test it tomorrow at work, I've got a handful of Win2K8 workstations with 108GB of RAM available.

AiZ

[byronknoll]

Bad news. I was not able to create the compile which gives the same size on enwik6 as stated at cmix site (http://www.byronknoll.com/cmix.html)
The size given there is 181908 bytes. I got 181915 or 181917 or even 181929. Seems compiler optimization options have effect on resulting size.
Currently I don't know how to achieve the same one.

P.S. My results are with -c english.dic enwik6 out.cmix

AiZ, your compile crashes on my system.

Ah, I wasn't expecting it to get exactly the same output as the linux version - as you said the compiler optimization options can change the output. However, it should perform approximately as well. Skymmer, could you post your cmix.exe compile, since the one I posted earlier doesn't seem to work for other people? I would be curious to see how it does on enwik8 (with a decompression test as well to verify it successfully reproduces the original file).

[Skymmer]

Skymmer, could you post your cmix.exe compile

Sure, here it is.

[Darek]

Now it works. Thank you Byron Skymmer!

I'll test enwik8.

After first short tests - this version is a little bit better than your latest compile from 4th of November - in average by 0,004% - differences are in bytes. Some files are comresssed worse, most - better.

[AiZ]

Hello,

FYI, with Skymmer version

D:\Tmp\cmix>timer64 cmix -c english.dic enwik8 enwik8.paq

 cmix-v6-win64 build: 20141208 21:23
 Number of models:      1728
 Num. of mixer neurons: 414264
100000000 bytes -> 15776674 bytes in 45208.29 s.
cross entropy: 1.262


Kernel  Time =    21.044 =    0%
User    Time = 45199.792 =   99%
Process Time = 45220.836 =   99%    Virtual  Memory =  28427 MB
Global  Time = 45225.501 =  100%    Physical Memory =  18236 MB

and, when compiled with 64bit g++.exe (Rev2, Built by MSYS2 project) 4.9.2

Program received signal SIGSEGV, Segmentation fault.
0x000000000041da0d in (anonymous namespace)::BH<4>::operator[] (
    this=0xe2a1a0 <_ZZN12_GLOBAL__N_113contextModel2EvE4rcm7>, i=0)
    at src/models/paq8hp.cpp:689
689         if (p[2]==0) { *(U16*)p=chk; break; }

AiZ

[byronknoll]

Thanks Skymmer/AiZ/Darek for the testing. It looks like my fix was unsuccessful - the Windows compile should be getting 15,73x,xxx bytes on enwik8. It is hard to debug the issue when I can't test on my own computer! I will keep working on it.

[Darek]

Enwik8 test with dictionary option:

compressed size: 15'776'674 bytes = 0,01% better than previous compile.
compression time: 31'140s = 10% faster than previous compile.
decompression time: 30'236s. Checksum verified, ok.

That exact as AiZ scores.

Darek

p.s. hmmmm, one question - which dictionary will you use to test? I've used cmix6 dictionary. Maybe some differences in this file could provide to different scores.

[byronknoll]

p.s. hmmmm, one question - which dictionary will you use to test? I've used cmix6 dictionary. Maybe some differences in this file could provide to different scores.

Yeah you should use the cmix6 dictionary (that is what I use for all benchmarks).

[byronknoll]

Good news. I upgraded to "Windows 7 Ultimate", so now I have enough available RAM to test cmix. I tried testing my compile above on enwik8:

$./cmix -c dictionary/english.dic data/enwik8 data/out.dat
100000000 bytes -> 15738924 bytes in 30872.32 s.
cross entropy: 1.259

Two bytes off from the Linux version (which was 15738922). So it looks like compiling (with my fix to paq8hp.cpp) in cygwin with g++ works.

[Darek]

Byron: Then we could find how upgrade system to yours cygwin library.

Ok, I've installed cygwin from this link: http://www.cygwinports.com/

and it works, however for dictionary option for most files I've got error:

0 [main] cmix6 5816 cygwin_exception:pen_stackdumpfile: Dumping stack trace to cmix6.exe.stackdump

Maybe it's physical memory problem - I've got only 16GB of ram. However Skymmer compiles doesn't crush and works with swap file correctly.

I've tested this compile on my testbed files and compare to previous builds (all in nondictionary mode) and in overall it's compression ratio is about 0,014% worse than Skymmer compile from 9.12.2014. Similar decrease of compression ratio I've seen on few files which run with dictionary option.

Maybe big text files with dictionary option are compress better. I can't test enwik8 file...

Darek

[Matt Mahoney]

Brian Knoll released cmix v8. http://www.byronknoll.com/cmix.html
It already had the number 1 spot on both LTCB and Silesia. The new version improves compression even further.
However, it takes 4 days to compress or decompress enwik9 on a 32 GB i7-3770.
His test results are as follows:

Version: 8
Compressed size of enwik8: 15709216 bytes
Compressed size of enwik9: 123930173 bytes
The program doesn't contain any configurable options.

Size of source code as a zip file: 164,882 bytes
zip file contains:
- all source code
- makefile
- dictionary (compressed using cmix from 465211 bytes -> 89713 bytes)

enwik9 compression time: 344244.32 seconds
enwik9 decompression time: 346641.12 seconds
Approximate memory used: 30311032 KiB

Description of test machine:
processor: Intel Core i7-3770K
memory: 31.4 GiB
OS: Linux Mint 17

Description and source code: http://www.byronknoll.com/cmix.html

Silesia compressed size in bytes:

dicke: 1893344
mozil: 9565086
mr: 2020572
nci: 838631
ooff: 1338376
osdb: 2020534
reym: 770535
samba: 2591823
sao: 3767137
webst: 4689079
x-ray: 3568206
xml: 244270

total: 33307593

[Darek]

Brian Knoll released cmix v8. http://www.byronknoll.com/cmix.html

Matt,
there are small typo in samba score on Silesia test scores page - now the score is 2951 instead 2591.

Question/request - if anyone could made Win64 build of this version of CMIX?

Darek

[przemoc]

Question/request - if anyone could made Win64 build of this version of CMIX?

I've built it in MSYS2's MinGW-w64 Win64 Shell using recent gcc version 5.2.0 (Rev4, Built by MSYS2 project) with

all: LFLAGS += -static -static-libgcc -static-libstdc++

added to Makefile to include non-win dependencies. The thing is I cannot test whether this build is good, because my notebook quickly runs out of memory...

[Darek]

At first very thanks for this compile.

However it doesn't work in my computer.

I've built it in MSYS2's MinGW-w64 Win64 Shell using recent gcc version 5.2.0 (Rev4, Built by MSYS2 project)

I've already installed older version of MiniGW and probably due to this program crashes and calls:

"
terminate called after throwing an instance of 'char const*'
This application has requested the Runtime to terminate it in an unusual way. Please contact the application's suport team for more information.
"

Where I could find and download proper version of MiniGW-w64?

Darek

[byronknoll]

I have tried compiling cmix for Windows using both cygwin and mingw. Compiling with cygwin should work without any problems. However, it is not convenient to widely distribute that binary since it is incompatible with other versions of cygwin. I haven't been able to get cmix successfully compiled using mingw. Looking at older messages in this thread I see Skymmer compiled a version that works: "cmix-v6-win64-fixed-paq8hp-model.7z". I wonder how Skymmer compiled it. It would be great if I could include an official Windows binary along with the cmix releases.

[load]

i compiled it too for windows, but it crashes as well. maybe not enough memory?

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc

/edit : attached another version, with only -O3 instead of -Ofast, still nock luck here.

i will try on my desktop pc later, with 32gb ram.

[Darek]

I Looking at older messages in this thread I see Skymmer compiled a version that works: "cmix-v6-win64-fixed-paq8hp-model.7z". I wonder how Skymmer compiled it.

For my computer setup Skymmer version was the best and the most stable at all. It won't crashes nor hangs. Other versions have problems with dictionary option for many files...

Maybe it's worth to mention that I have two laptops - with 16GB and 32GB of memory, however this latest compile (as well as not Skymmer's builds) crashes on both machines. From other side Skymmer's version works fine on both machines as well - even on 16GB computer. Of course 16GB machine was much slower due to swap memory usage, but I've tested all files w/o any issues on this laptop too.

Then in my opinion it's not directly memory problem - maybe some issues with memory allocations?

Darek

@load: - I've the same error as you with your compile but w/o part of message about memory allocation:"terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc"

[Darek]

Request for Skymmer: could you try to compile your build of the latest version od CMIX (Ver.8 )?

Darek

[przemoc]

At first very thanks for this compile.

However it doesn't work in my computer.

Well, maybe this build is dysfunctional. As I wrote earlier, I quickly get:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

when trying to compress something.

I've already installed older version of MiniGW and probably due to this program crashes and calls:

"
terminate called after throwing an instance of 'char const*'
This application has requested the Runtime to terminate it in an unusual way. Please contact the application's suport team for more information.
"

Your already installed MinGW shouldn't matter here, because I built the exe statically (by modifying Makefile as I wrote earlier).

Where I could find and download proper version of MiniGW-w64?

MinGW-w64 has many different builds. Bare compiler is not that convenient to use if you want to build POSIX-y applications, requiring some basic UNIX utilities (like make, awk, sed, grep, rm, etc.).

If you want native windows build with basic stuff ready (or easily ready) to use, then MSYS2 is nicely wrapped bootstrapping environment, where you can easily download most stuff already prebuilt.
Another nice MinGW-w64 distro is done by STL. Nowadays it's MSYS2-based too (EDIT: or at least it's built using MSYS2 according to its page, haven't used it myself for a few years). It has even git within it, but there is git-less installer too.

For cross-compiling from Linux in the past I was using Ruben's builds (he no longer provides new versions) and Win-builds (previously known as yypkg mingw-builds). Nowadays many popular distributions have mingw-w64 in their repositories.

[Skymmer]

Well, I'm trying to get the working binary all this week and I'm almost ready to give up. God damned SIGSEGV is back again. Debugger says:
Program received signal SIGSEGV, Segmentation fault.
0x000000007701f280 in ntdll!RtlIsValidIndexHandle () from C:\Windows\SYSTEM32\ntdll.dll

Probably its possible to find out more but I'm simply don't have any debugging skills to make it right.
I have one theory why we have problems compiling cmix on Windows but its only an idea.
Shortly: LP64 data model vs. LLP64 one. More info by the links below:
https://cygwin.com/cygwin-ug-net/pro...ng.html#gcc-64
http://stackoverflow.com/questions/7...ng-in-64-bit-c
https://en.wikipedia.org/wiki/64-bit...it_data_models

Anyway its nice to see that cmix development goes on.

[byronknoll]

Hi Skymmer, can you give more information on how you compiled "cmix-v6-win64-fixed-paq8hp-model.7z"? Which compiler did you use? Thanks!

[Skymmer]

Actually its based on fix provided by you in post#31 in this thread. Plus some modifications.

predictor.cpp
added #include <stdio.h>

runner.cpp
added #include <cstdlib>

paq8hp.cpp
changed U8 *p=&t[i*B-B]; to U8 *p=&t[i*B] - B;
changed throw message; to printf("%s",message);

paq8l.cpp
line 56: changed #ifndef WINDOWS to #ifndef min
changed throw message; to printf("%s",message);

Compiler: Nuwen 12.2 MinGW Distro (GCC 4.9.2 based)
nuwen.net

[Darek]

@Przemoc - thank you for full description. I'll try to run this biuld on other computers. Maybe it will works...

@Skymmer - also thank you for effort to made biuld. As it looks all Cmix versions were hard to compile and made proper working exec on Windows.

[Darek]

@Skymmer - have you now stable build of cmix8 for win64?