cmix

[byronknoll]

In case it is useful for making a Windows build, the cmix code at https://github.com/byronknoll/cmix has some bug fixes. The current github code will be the cmix v9 release (assuming all my benchmarks validate successfully). It will take a week or two until the official v9 release. enwik8 will have a compressed size of 15615723.

[load]

the one from github still crashes under windows. maybe someone else will try.

[byronknoll]

the one from github still crashes under windows. maybe someone else will try.

@load, what did you use to compile it? I expect it should work in Windows compiled in cygwin with g++.

[load]

cygwin is what i used. but i also tried now some msys with the same result.

i686-w64-mingw32-g++
x86_64-w64-mingw32-g++

the 32bit crashed with the same error message. it seems cmix wants to allocate an enormous amount of memory before it crashes... windows even needs to use the disk swap file, although the file i wanted to compress is only a few bytes.

[Darek]

Skymmer has made very stable version of CMIX V6

[load]

i know. but he refuses to tell us what he changed in the source, what he obviously. did.

[Darek]

i know. but he refuses to tell us what he changed in the source, what he obviously. did.

Fact is that Skymmer version of CMIX6 has slightly better compression than last stable Byron CMIX6 build - however Byron tells us that this compile could have slightly worse compression ratio than Linux version.

[schnaader]

it seems cmix wants to allocate an enormous amount of memory before it crashes... windows even needs to use the disk swap file, although the file i wanted to compress is only a few bytes.

Please make clear what "enormous" means here. Right from the cmix website:

At least 32GB of RAM is recommended to run cmix.

Corpus	Original size (bytes)	Memory usage (KiB)	Compressed size (bytes)		Compression time (seconds)	

enwik6	1000000	                19470840                181738                          561.83

So even for a 1 MB file, you'll need 19 GB RAM.

[Darek]

Please make clear what "enormous" means here. Right from the cmix website:

Corpus	Original size (bytes)	Memory usage (KiB)	Compressed size (bytes)		Compression time (seconds)	

enwik6	1000000	                19470840                181738                          561.83

So even for a 1 MB file, you'll need 19 GB RAM.

I've got laptop with 32GB of ram and cmix7 and cmix8 compiles crashes...
From other side CMIX6 versions (both Byron and Skymmer) runs fine with my other laptop with "only" 16GB by heavily use of swap file.

[byronknoll]

i know. but he refuses to tell us what he changed in the source, what he obviously. did.

Skymmer posted the steps he used to make the compile in post#58 of this thread. After my benchmarks finish running (1-2 weeks), I will try adding those changes into cmix to try to improve Windows compatibility. I will also try posting a cmix v9 executable to this thread to see if it works on other people's computers.

[load]

Skymmer posted the steps he used to make the compile in post#58 of this thread. After my benchmarks finish running (1-2 weeks), I will try adding those changes into cmix to try to improve Windows compatibility. I will also try posting a cmix v9 executable to this thread to see if it works on other people's computers.

i did read that, and applied this to cmix6, but still it crashed with exactly the same error.

i rebuilt cmix8 again (msys/stable gcc 5.2) with "fixes" from #58 and tried to run it on my desktop system with 32gb ram and only 1.5gb in use, same crash.
it can't be the lack of memory, as it eats only ~10gb ram when i get the crash dialog.


maybe i'm doing something wrong (i'm not really good at this), but i don't see what could be done wrong, honestly. it's a straight forward "make", and no errors or warnings turn up during compiling process.
the only thing i added is -static-libgcc -static-libstdc++ to the linker flags, for obvious reasons.

[byronknoll]

I added Skymmer's fixes to the code on github.

I got a chance to try compiling cmix in Windows. cmix v9 seemed to work normally with cygwin/g++. I am using cygwin 1.7.30.

I also tried compiling using mingw 13.4 from here: http://nuwen.net/mingw.html
cmix v9 crashed with a segmentation fault. The crash seems to be coming from the PAQ8L model. When I change the memory parameter of PAQ8L from 11 to 10 (predictor.cpp line 97), the compile no longer crashes. However, this probably will hurt compression rate, so I will try to find a better fix.

I am attaching the binaries that I built. Can someone try testing them? The cygwin executable might only work in cygwin 1.7.30.

[Darek]

I added Skymmer's fixes to the code on github.

I got a chance to try compiling cmix in Windows. cmix v9 seemed to work normally with cygwin/g++. I am using cygwin 1.7.30.

I also tried compiling using mingw 13.4 from here: http://nuwen.net/mingw.html
cmix v9 crashed with a segmentation fault. The crash seems to be coming from the PAQ8L model. When I change the memory parameter of PAQ8L from 11 to 10 (predictor.cpp line 97), the compile no longer crashes. However, this probably will hurt compression rate, so I will try to find a better fix.

I am attaching the binaries that I built. Can someone try testing them? The cygwin executable might only work in cygwin 1.7.30.

The mingw version runs fine on 32GB, I've started to test it. On 16GB doesn't work at all.
The cygwin version gives error as on screen. That means "The application was improperly launched (0xc000007b). Click OK to close it.", however as you wrote my cygwin1.dll is 1.5.18.

[comp1]

The minigw version runs fine on 32GB, I've started to test it. On 16GB doesn't work at all.
The cygwin version gives error as on screen. That means "The application was improperly launched (0xc000007b). Click OK to close it.", however as you wrote my cygwin1.dll is 1.5.18. Where I can download 1.7.30 version?

I also got the same results. Error with cygwin version and no problems with MinGW version on my 32GB machine.

[Darek]

I also got the same results. Error with cygwin version and no problems with MinGW version on my 32GB machine.

Ok, in attached file you find three dll needed to run cygwin version of CMIX V9 on Win64. Just put it into Cmix directory. Now CygWin version also run!

Darek

EDIT:

On my 16GB laptop with maximum swap file, and free memory b0th versions running too.

But is something strange that I've found. Mingw version on 32GB laptop gives slightly better scores than on 16GB - couple of bytes (1 up to 9). I've checked names and are identical, theoretical memory usage are the same - both 32GB, but scores are different. Is possible that something depends on memory amounts allocated - which could be a little different?

[load]

cygwin crashes here, no matter what dll i use:

Exception: STATUS_ACCESS_VIOLATION at rip=00076F6F645
rax=00000006B1834CA0 rbx=00000006B1834CA0 rcx=0000000771840020
rdx=0000000000000000 rsi=00000000C0000000 rdi=0000000000000000
r8 =0000000000000000 r9 =03FFFFFFFBFFFD33 r10=00000006B1834C90
r11=0000000000000000 r12=00000006B17BEAB0 r13=0000000000008000
r14=00000006B17BE8A0 r15=0000000000008000
rbp=0000000000000000 rsp=0000000000229C88
program=d:\\cmix.exe, pid 5568, thread main
cs=0033 ds=002B es=002B fs=0053 gs=002B ss=002B
Stack trace:
Frame Function Args
00000000000 00076F6F645 (00000000000, 000FFFFFFFF, 00000000000, 006B1834CA0)
00000000000 0007759BA46 (00000000000, 00000000000, 006B17BEAB0, 006B1834CA0)
00000000000 001800C4917 (00000000000, 00000000000, 001801360DF, 00000000000)
00000000000 001801451FB (0000000001A, 00000000001, 00180111C9B, 0000000001A)
00000000000 001800C00EF (0018013A039, 006000F70D0, 00180139FF8, 00600000020)
00000000000 00180111C9B (006000F70D0, 00180139FF8, 00600000020, 00000000030)
00000000000 001004136C0 (00180139FF8, 00600000020, 00000000030, 002000F491
00000000000 0018013A039 (00180139FF8, 00600000020, 00000000030, 002000F491
End of stack trace

mingw runs fine.also when i compile it myself using the predictor.cpp changes.

[byronknoll]

But is something strange that I've found. Mingw version on 32GB laptop gives slightly better scores than on 16GB - couple of bytes (1 up to 9). I've checked names and are identical, theoretical memory usage are the same - both 32GB, but scores are different. Is possible that something depends on memory amounts allocated - which could be a little different?

Hmm, that is strange. No, the amount of available memory shouldn't change the compression rate. Maybe this is due to compiling with -Ofast, which makes it faster but not standards compliant. Perhaps that makes it not portable between computers.

[Darek]

Hmm, that is strange. No, the amount of available memory shouldn't change the compression rate. Maybe this is due to compiling with -Ofast, which makes it faster but not standards compliant. Perhaps that makes it not portable between computers.

I confirm - I've tested some files with different memory amount on one computer and scores are identical. Something is curious between my two computers.

@Load - did you try my "cmixV9_cygwin_dll.7z" set? It works for my both computers.

[byronknoll]

@Darek, I am attaching a mingw compile which is slower but hopefully should give the same output on any computer. Could you try testing the following executable on your two computers to see if they give the same output? Thanks!

[Darek]

@Darek, I am attaching a mingw compile which is slower but hopefully should give the same output on any computer. Could you try testing the following executable on your two computers to see if they give the same output? Thanks!

Yes. I'm out of home now. I'll test It as I'll arrive Home tommorow.

[Sportman]

I am attaching the binaries that I built. Can someone try testing them?

cmix9 under Windows command line, no dictionary:
enwik8 100,000,000 bytes to 16,270,942 bytes in 56825.94 sec.

[Darek]

@Darek, I am attaching a mingw compile which is slower but hopefully should give the same output on any computer. Could you try testing the following executable on your two computers to see if they give the same output? Thanks!

Now it's OK, both computers give the same results on MinGw version.
CygWin version is OK too.

Why the results are different on different machines on first compilation mingw?

[byronknoll]

Linux, dictionary preprocessing, PAQ8L memory parameter set to 11: enwik8 compresses to 15,615,723 in 33,175.59 seconds with 26,313,052 KiB memory
Linux, dictionary preprocessing, PAQ8L memory parameter set to 10: enwik8 compresses to 15,627,536 in 33,244.92 seconds with 23,046,324 KiB memory

I haven't been able to figure out how to fix the mingw compile without setting PAQ8L to 10. So, I think I will just make 10 the default for cmix v9 to improve Windows compatibility. This also reduces memory usage by about 3GB (which I will probably make use of for cmix v10).

[byronknoll]

Why the results are different on different machines on first compilation mingw?

For the second mingw compile, I removed -march=native and set -O3 instead of -Ofast.

[Darek]

For the second mingw compile, I removed -march=native and set -O3 instead of -Ofast.

Yes, I understand it, but the question is how this change could made one executable which gives different scores on different machines.

I'm still testing. It's going slowly due to:
a) use swap file which slow down compression performance,
b) using swap file makes CPU usage down which provides CPU booster to think that we aren't using much procesor power and didn't overclock procesor or even set clock down. Instead 3.8GHz I've running on 2.6GHz.

One insight from testing and CMIX crashing because of memory usage: all crashes are on the same 14 files from my testbed - any version, even older gets crash on the same files. Size is not an factor because it's on small and bigger files, but it's always occurs when I run dictionary usage and CMIX started to pretraining. For Cygwin version it occurs also when I'm running on 32GB computer. Errors are similar: "0 [main] cmixX XXXX cygwin_exceptionen_stackdumpfile: Dumping stack trace to cmixX.exe.stackdump: . That means the error is showing on pretraining stage, not in compression phase at all.

Darek

[byronknoll]

Yes, I understand it, but the question is how this change could made one executable which gives different scores on different machines.

I'm not exactly sure. When using -Ofast it is no longer standards compliant, so for example some floating point math can give different output than -O3. I think different computer architectures can also cause the output to change.

One insight from testing and CMIX crashing because of memory usage: all crashes are on the same 14 files from my testbed - any version, even older gets crash on the same files. Size is not an factor because it's on small and bigger files, but it's always occurs when I run dictionary usage and CMIX started to pretraining. For Cygwin version it occurs also when I'm running on 32GB computer. Errors are similar: "0 [main] cmixX XXXX cygwin_exceptionen_stackdumpfile: Dumping stack trace to cmixX.exe.stackdump: . That means the error is showing on pretraining stage, not in compression phase at all.

This sounds like a bug - can you give an example file which causes cmix v9 to crash?

[Darek]

This sounds like a bug - can you give an example file which causes cmix v9 to crash?

Please, example in attached zip file.

[byronknoll]

Please, example in attached zip file.

Thanks, I have fixed this bug. I am attaching a new executable with the fix. The fix is also submitted to Github.

The bug only affected the mingw build. I was using "tmpfile()", which apparently needs administrator privileges to work in Windows. I changed the code to write temporary files to the output directory rather than using tmpfile().

[Darek]

The bug only affected the mingw build.

Thanks for this version, now mingw is stable at all on 16GB and 32GB computer!

But, this mentioned error ""0 [main] cmixX XXXX cygwin_exceptionen_stackdumpfile: Dumping stack trace to cmixX.exe.stackdump:" goes from cygwin version. Maybe it's different bug. For this version and these 14 files error ocurs both for 16GB and 32GB computer. As I mentioned only when CMIX start to pretraining.

[byronknoll]

But, this mentioned error ""0 [main] cmixX XXXX cygwin_exceptionen_stackdumpfile: Dumping stack trace to cmixX.exe.stackdump:" goes from cygwin version. Maybe it's different bug. For this version and these 14 files error ocurs both for 16GB and 32GB computer. As I mentioned only when CMIX start to pretraining.

Ah, I guess the tmpfile problem can affect the cygwin version too. Potential fixes:
1) Run cmix with administrator privileges.
2) Run cmix within the cygwin environment.
3) Compile a new version of cmix in cygwin, using the fixed code on github. I am running enwik9 in Linux (I have one computer with dual boot), so I won't be able to make a new cygwin compile for a while.