cmix

**Shelwien** · 1st January 2020, 23:07

100000000/126675.67 = 789 bytes/s. But that's still 10x faster than NNCP :)

**byronknoll** · 2nd January 2020, 10:19

Originally Posted by encode

Results on my PC:
CPU: Intel Core i5-9600K @ 4.8 GHz
MoBo: MSI Z390 GAMING EDGE
RAM: HyperX Predator 32 GB (2x16) 3200 MHz CL16
Storage: Samsung 970 EVO Plus 250 GB M.2 NVMe SSD

Code:

C:\cmix>timer cmix -c enwik8 enwik8.cmix18

Timer 3.01  Copyright (c) 2002-2003 Igor Pavlov  2003-07-10
100000000 bytes -> 15384933 bytes in 126675.67 s.
cross entropy: 1.231

Kernel Time  =    12.140 = 00:00:12.140 =   0%
User Time    = 126660.828 = 35:11:00.828 =  99%
Process Time = 126672.968 = 35:11:12.968 =  99%
Global Time  = 126675.921 = 35:11:15.921 = 100%

ENWIK9 results will take some time...

I recommend enabling dictionary preprocessing to improve compression rate and compression time: timer cmix -c dictionary/english.dic enwik8 enwik8.cmix18
Compiling cmix yourself will also produce a faster executable. For cmix v18 on my computer, enwik8 compresses to 14838332 in 57508 seconds.

**Darek** · 3rd January 2020, 02:44

>Compiling cmix yourself will also produce a faster executable. For cmix v18 on my computer, enwik8 compresses to 14838332 in 57508 seconds.
@Byron -> how I can compile cimix by myself? Is there any fast and easy comiler to use for it? I'm a lamer in this kind of topics.

**Shelwien** · 3rd January 2020, 10:23

For windows you can get gcc/mingw: https://sourceforge.net/projects/min...onal%20Builds/
Then compile it like this: https://encode.su/threads/1925-cmix?...ll=1#post62052
There would be more compiler options for best speed, though.

**byronknoll** · 3rd January 2020, 10:24

Originally Posted by Darek

>Compiling cmix yourself will also produce a faster executable. For cmix v18 on my computer, enwik8 compresses to 14838332 in 57508 seconds.
@Byron -> how I can compile cimix by myself? Is there any fast and easy comiler to use for it? I'm a lamer in this kind of topics.

In Linux, just run "make". In Windows it is a bit more difficult. You can try either MinGW (http://nuwen.net/mingw.html) or Cygwin (https://www.cygwin.com) and then run "make".

**Shelwien** · 3rd January 2020, 10:55

For "make" its better to get msys: http://www.msys2.org/ - its the only system with a working package manager.
Mingw distributions frequently don't have any make at all, or it doesn't work.
And cygwin requires manually selecting some packages in GUI setup - you have to know what you need when installing it.

So I think the best method is mingw + use a list file for g++ @list, like I suggested before. Make is not needed to build cmix.

**encode** · 3rd January 2020, 11:26

It's an executable from your webpage. Now I'm running the ENWIK9 compression - I think it will take more than 10 days. Will test the dictiobary right after!

**hexagone** · 3rd January 2020, 22:05

Mingw-w64 (x86_64-8.1.0-posix-seh-rt_v6-rev0) works well for me. Just replace clang++ with g++ in the makefile and run "mingw32-make".

**bwt** · 4th January 2020, 16:04

Originally Posted by byronknoll

I recommend enabling dictionary preprocessing to improve compression rate and compression time: timer cmix -c dictionary/english.dic enwik8 enwik8.cmix18
Compiling cmix yourself will also produce a faster executable. For cmix v18 on my computer, enwik8 compresses to 14838332 in 57508 seconds.

The different between encode and uour reseult is 546601 bytes. is it possible to reach that by use dictionary preprocessing ? i gues although use dictionary the result still different.

**encode** · 6th January 2020, 17:45

ENWIK9 compression's running for 6 days = 34% complete. Has anyone rechecked the results ever?

**suryakandau@yahoo.co.id** · 9th January 2020, 03:22

Originally Posted by encode

ENWIK9 compression's running for 6 days = 34% complete. Has anyone rechecked the results ever?

Has anyone rechecked the nncp result too ?

**encode** · 13th January 2020, 13:39

Results with a slightly higher overclock:
i5-9600K @ 4.9 GHz all cores. 5.0 GHz and above requires much higher voltage which is not safe for 24/7 running.

Code:

C:\cmix>timer cmix -c dictionary/english.dic enwik8 enwik8.cmix18

Timer 3.01  Copyright (c) 2002-2003 Igor Pavlov  2003-07-10
100000000 bytes -> 14847703 bytes in 76734.24 s.
cross entropy: 1.188

Kernel Time  =    15.625 = 00:00:15.625 =   0%
User Time    = 76713.015 = 21:18:33.015 =  99%
Process Time = 76728.640 = 21:18:48.640 =  99%
Global Time  = 76736.093 = 21:18:56.093 = 100%

And I don't believe your compression/decompression timings. Or it is an extremely poor EXE compile at your website - please recompile and update the page!

**Shelwien** · 13th January 2020, 14:13

cmix dynamically allocates lots of small memory blocks (mostly for LSTM model),
its known to have a large effect on speed and memory consumption depending on OS and compiler.
Also there could be a difference in vector extensions used.

**byronknoll** · 13th January 2020, 21:34

Originally Posted by encode

And I don't believe your compression/decompression timings. Or it is an extremely poor EXE compile at your website - please recompile and update the page!

The executable on the cmix page was compiled without "-march=native" to improve compatibility between computers. If you compile your own executable with "-Ofast -march=native" it will be significantly faster. I think the main reason is auto-vectorization and SIMD. The cmix benchmarks I ran were not with the public executable, but with one compiled with "-march=native".

**encode** · 14th January 2020, 18:26

Results with a compile by Shelwien (Thank you!)

Code:

C:\cmix>timer cmix -c dictionary/english.dic enwik8 enwik8.cmix18

Timer 3.01  Copyright (c) 2002-2003 Igor Pavlov  2003-07-10
100000000 bytes -> 14846066 bytes in 63749.76 s.
cross entropy: 1.188

Kernel Time  =    13.484 = 00:00:13.484 =   0%
User Time    = 63731.453 = 17:42:11.453 =  99%
Process Time = 63744.937 = 17:42:24.937 =  99%
Global Time  = 63751.796 = 17:42:31.796 = 100%

**Sportman** · 27th February 2020, 21:32

cmix (v18 ) -c english.dic enwik9:
115,739,547 bytes, 756,510.756 sec. (8.76 days, 25GB memory use, cross entropy 0.926)

**michael maniscalco** · 28th February 2020, 07:59

Originally Posted by Sportman

cmix (v18 ) -c english.dic enwik9:
115,739,547 bytes, 756,510.756 sec. (8.76 days, 25GB memory use, cross entropy 0.926)

8.76 days ... think about that (for 8.76 days).

**lzhuff** · 29th February 2020, 03:34

[QUOTE=Sportman;64022]cmix (v18 ) -c english.dic enwik9:
115,739,547 bytes, 756,510.756 sec. (8.76 days, 25GB memory use, cross entropy 0.926)[/Q

For enwik10 it needs 87.6 days (~3 months)

**bwt** · 29th February 2020, 03:36

[QUOTE=lzhuff;64042]

Originally Posted by Sportman

cmix (v18 ) -c english.dic enwik9:
115,739,547 bytes, 756,510.756 sec. (8.76 days, 25GB memory use, cross entropy 0.926)[/Q

For enwik10 it needs 87.6 days (~3 months)

**suryakandau@yahoo.co.id** · 3rd March 2020, 15:43

[QUOTE=lzhuff;64042]

Originally Posted by Sportman

cmix (v18 ) -c english.dic enwik9:
115,739,547 bytes, 756,510.756 sec. (8.76 days, 25GB memory use, cross entropy 0.926)[/Q

For enwik10 it needs 87.6 days (~3 months)

Which part of cmix that reduced can be run under 8gb ram ? Can reduced paq n paq8h level from 11 to six can run under 8gb ram ?

**suryakandau@yahoo.co.id** · 16th March 2020, 19:39

Originally Posted by Sportman

cmix (v18 ) -c english.dic enwik9:
115,739,547 bytes, 756,510.756 sec. (8.76 days, 25GB memory use, cross entropy 0.926)

@sportman could you test the decompression using cmix ? I have tested cmix v17 with reduced paq8hp n paq8 level to 4 and the checksum value don't match

**suryakandau@yahoo.co.id** · 16th March 2020, 19:43

@sportman If you want test the decompression function of cmix, use small file, it does not take very long time

**suryakandau@yahoo.co.id** · 2nd April 2020, 04:20

i have tested cmix17 on wrtpre.cpp and the hash value after decompression is not match with the original file. why ???
this is the hash value of wrtpre.cpp 3FE3BD3E77A2A34869EC12FD77491EF9D0192BFA
i attach the source code and the binary of cmix17 and compiled it using dev c++

**byronknoll** · 2nd April 2020, 20:35

Originally Posted by suryakandau@yahoo.co.id

i have tested cmix17 on wrtpre.cpp and the hash value after decompression is not match with the original file. why ???
this is the hash value of wrtpre.cpp 3FE3BD3E77A2A34869EC12FD77491EF9D0192BFA
i attach the source code and the binary of cmix17 and compiled it using dev c++

I might be able to help debug this. Did you make any modifications to cmix v17? Did you use a dictionary when compressing and decompressing? Can you post a copy of wrtpre.cpp so I can try reproducing the problem on my computer?

**suryakandau@yahoo.co.id** · 2nd April 2020, 23:59

Originally Posted by byronknoll

I might be able to help debug this. Did you make any modifications to cmix v17? Did you use a dictionary when compressing and decompressing? Can you post a copy of wrtpre.cpp so I can try reproducing the problem on my computer?

i just setting paq8 and paq8hp to level 6 on cmix17. i do not use dictionary when compress and decompress. here is wrtpre.cpp

**byronknoll** · 3rd April 2020, 09:53

It works for me. Here is a Colab where you can see it compress+decompress to the same md5: https://colab.research.google.com/dr...eX-ZSjMgF29Ci7

**suryakandau@yahoo.co.id** · 3rd April 2020, 12:25

Originally Posted by byronknoll

It works for me. Here is a Colab where you can see it compress+decompress to the same md5: https://colab.research.google.com/dr...eX-ZSjMgF29Ci7

How to use colab ?

**suryakandau@yahoo.co.id** · 3rd April 2020, 12:59

Originally Posted by suryakandau@yahoo.co.id

How to use colab ?

Do you use my binary compiled with Dev c++ ??

**suryakandau@yahoo.co.id** · 3rd April 2020, 15:45

Originally Posted by suryakandau@yahoo.co.id

Do you use my binary compiled with Dev c++ ??

my assumption is you have compiled the source code on linux...the question is why the same source code compiled with different compiler can cause the decompress file corrupted

**byronknoll** · 4th April 2020, 01:11

Originally Posted by suryakandau@yahoo.co.id

my assumption is you have compiled the source code on linux...the question is why the same source code compiled with different compiler can cause the decompress file corrupted

I didn't use your binary. Here are some suggestions that might help:
- change the compiler flag from -Ofast to -O3. The binary will be slower, but might fix the issue you are seeing.
- upgrade your compiler to a more recent version.
- change to a different compiler - I recommend clang.

Thread: cmix

Thread Tools

Search Thread

Display

Thanks:

Thanks:

Thanks:

Tags for this Thread

Posting Permissions