cmix

[Darek]

Yes. Sorry for mistake.
From other hand I'm still testing rnn for cmv for first three files.
Rsults are interesting but it's still few days to finish.

[moisesmcardona]

Can someone compile the cmix v17 build for Windows? https://github.com/byronknoll/cmix/releases/tag/v17

[Shelwien]

Build target x64/SSE4. Not tested. IntelC19 and gcc82/mingw.

[Darek]

Can someone compile the cmix v17 build for Windows? https://github.com/byronknoll/cmix/releases/tag/v17

There is Byron's Windows compile on official cmix page and it works fine: http://www.byronknoll.com/cmix.html

[Darek]

Byron - are yours test scores on official cmix page made with new dictionary file included in cmix_v17 package?

[byronknoll]

Byron - are yours test scores on official cmix page made with new dictionary file included in cmix_v17 package?

Yes.

[Darek]

Here are scores of my testset for cmix v17 vs. latest cmix v16f. In total there is a 8450 bytes loss, mainly for biggest, non-textual files.
Pure textual files got very nice gain = 0.35% which is impressive.

p.s. scores aren't optimal yet - I've based on v16e/f optimal options.
I'm testing now other options (there are four: 1) just -c, 2) -c english.dic, 3) -s english.dic then -c and 4) -s english.dic then -c english.dic).

I'll try to test v17 with older dictionary (from v16).

[Darek]

Build target x64/SSE4. Not tested. IntelC19 and gcc82/mingw.

IC19 version doesn't work on my system.

Comparison of Byron's compile and gcc82 ( just -c option):
994350 bytes -> 139648 bytes in 1636.45 s. - Byron compile
994350 bytes -> 139799 bytes in 2737.96 s. - gcc82 compile

[Darek]

Final scores of my testset for cmix v17 - looks that this version have much different characteristics than v16f - different options got best scores.

@mpais - request - could you add LZW transform from paq8px v178 for TGA and TIFF files to cmix? Is it possible?

[Mauro Vezzosi]

Configurable adam learning rate, 2019/04/14:
- Why didn't you write directly learning_rate instead of learning_rate * 0.067?
- Will you use learning_rate statically (a fixed value) or dynamically (a value that changes during compression)?
- How about configuring also sqrt(5e-5 * t + 1)?
- In my tests sometimes beta1 = 0.7/0.8 is better than 0.9.
- https://bellard.org/nncp/nncp.pdf, 2.3 Training details: "We use the Adam optimizer [5] with beta1 = 0, beta2 = 0.9999 and eps = 10^-5. No gradient clipping is done.".

[byronknoll]

Configurable adam learning rate, 2019/04/14:
- Why didn't you write directly learning_rate instead of learning_rate * 0.067?

cmix actually uses two optimizers: gradient descent for the final softmax layer, and adam for everything else. My LSTM implementation takes a single learning rate parameter as input, which was used for the gradient descent (the adam parameters were just hardcoded in place to avoid too many configurable parameters). The latest commit is mostly refactoring: it uses the same learning rate parameter for both gradient descent and adam (with a multiplier in adam to make it close to the original hardcoded value).

- Will you use learning_rate statically (a fixed value) or dynamically (a value that changes during compression)?

"learning_rate_" is a fixed value (but "alpha" has the sqrt(5e-5 * t + 1) decay).

- How about configuring also sqrt(5e-5 * t + 1)?

I guess that could also be made into a configurable parameter - but I also want to avoid adding too many parameters to reduce complexity (i.e. only expose parameters that would make a significant difference when tuning for some new data set).

- In my tests sometimes beta1 = 0.7/0.8 is better than 0.9.
- https://bellard.org/nncp/nncp.pdf, 2.3 Training details: "We use the Adam optimizer [5] with beta1 = 0, beta2 = 0.9999 and eps = 10^-5. No gradient clipping is done.".

Yeah, I will continue experimenting with tuning these parameters. In NNCP the note is also added: "In our implementation, eps is added to the running average before taking the square root"

[byronknoll]

- https://bellard.org/nncp/nncp.pdf, 2.3 Training details: "We use the Adam optimizer [5] with beta1 = 0, beta2 = 0.9999 and eps = 10^-5

I tried using these parameters in cmix (and moving eps into the square root). It does significantly better on enwik8:

before: 14872664
after: 14856010

I'll continue doing some more parameter tuning.

[CompressMaster]

Stupid question, but I need to ask.
I need to compress files with latest cmix release (v17), but drag-and-drop method doesn´t work. What´s the correct command? I´m using Windows 7.

Thanks.
CompressMaster

[Shelwien]

\cmix\src>cmix.exe
cmix version 12
With preprocessing:
    compress:           cmix -c [dictionary] [input] [output]
    only preprocessing: cmix -s [dictionary] [input] [output]
    decompress:         cmix -d [dictionary] [input] [output]
Without preprocessing:
    compress:   cmix -c [input] [output]
    decompress: cmix -d [input] [output]

So either
cmix -c english.dic input output
or
cmix -c input output

Keep in mind that its good to have >32GB of RAM for it to work.

[CompressMaster]

I have encountered critical problem when I´ve tried to compress my textfile with cmix 17 (March 2019). Screenshot is attached. Seems to be an issue with memory (I have 4GB). Where´s the problem? Thanks.

CompressMaster

[byronknoll]

I have encountered critical problem when I´ve tried to compress my textfile with cmix 17 (March 2019). Screenshot is attached. Seems to be an issue with memory (I have 4GB). Where´s the problem? Thanks.
CompressMaster

Yup, that bad_alloc error is probably due to not enough memory. 32GB of RAM is recommended to run cmix. It might be possible to run with 4GB if you have a lot of swap space, but it would be very slow.

[Shelwien]

You can also compile it with -DDEFAULT_OPTION=4 or something.
Also this:

  AddByteModel(new PPMD::PPMD(6, 1200, manager_.bit_context_, vocab_));

  AddByteModel(new PPMD::PPMD(16, 1200, manager_.bit_context_, vocab_));

1200 here is allocated memory in MB, can be reduced too.
Maybe a few other places.

Of course, the compression would be worse than normal cmix.
But maybe still better than paq8?

[Mauro Vezzosi]

cmix v18 2019/08/01
http://www.byronknoll.com/cmix.html
https://github.com/byronknoll/cmix
https://github.com/byronknoll/cmix/releases
Changes from version 17 to version 18:
- LSTM improvements

@Byron: if it is not a problem, could you announce in this thread when you release a new version of cmix? TIA

[byronknoll]

@Byron: if it is not a problem, could you announce in this thread when you release a new version of cmix? TIA

Sure. Here are some results:

Compressed size of enwik8: 14838332 bytes
Compressed size of enwik9: 115714367 bytes


Size of source code as a zip file: 208,961 bytes
zip file contains:
- all source code
- makefile
- dictionary


enwik9 compression time: 602867.49 seconds
enwik9 decompression time: 601569.89 seconds
Approximate memory used: 25738196 KiB


Here are the Silesia results for "precomp v0.4.7 -cn | cmix v18"


dicke: 1813095
mozil: 6717412
mr: 1829883
nci: 792994
ooff: 1226244
osdb: 1962336
reym: 712062
samba: 1614935
sao: 3727061
webst: 4297002
x-ray: 3508509
xml: 236101


total: 28437634

[Darek]

Scores for my testset - 13KB of gain - nice, expecially due to "only" LSTM improvenments!
Good scores for bigger and exe files.

[Darek]

cmix v18 scores for 4 corpuses - mostly based on Byron's scores (thanks to post it on cmix page!).
In general for bigger coruses there are significant gain. For smaller files (Calgery, Canterbury) not so big...
One advantage over v17 version is a little sppedup (on my laptop) = compressing times of cmix v18 version are about 96% of cmix v17 - then there are good compression gain and a little sppedup in the same version!

In the second table there are the best scores for the 4 Corpuses on August 2019

[suryakandau@yahoo.co.id]

cmix v18 scores for 4 corpuses - mostly based on Byron's scores (thanks to post it on cmix page!).
In general for bigger coruses there are significant gain. For smaller files (Calgery, Canterbury) not so big...
One advantage over v17 version is a little sppedup (on my laptop) = compressing times of cmix v18 version are about 96% of cmix v17 - then there are good compression gain and a little sppedup in the same version!

In the second table there are the best scores for the 4 Corpuses on August 2019

How to compile cmix on devcpp++ ? I am interested in data compression too beside computer vision. This is my little project on computer vision under Android.
https://youtu.be/ofwEuCclswM

[Shelwien]

Just download http://www.byronknoll.com/cmix-v18.zip and compile normally?
It compiles with
find . -iname '*.cpp' >list
g++ @list
so just add all cpp files to a project or something?

[Gotty]

How to compile cmix on devcpp++ ?

devcpp is 4 years old.
I haven't tried, but I suspect it won't compile cmix out-of-the-box.
You should switch to a more recent development environment.

[Shelwien]

I tried and cmix18 compiles with gcc 5.10, but with "g++ -std=gnu++0x -static @list", there're errors without "-std"

[suryakandau@yahoo.co.id]

I tried and cmix18 compiles with gcc 5.10, but with "g++ -std=gnu++0x -static @list", there're errors without "-std"

i am using win7 64bit. i have copied cmix folder to drive c: and then rename makefile to makefile.bat and then run makefile.bat but still don't produce executable file

[Shelwien]

makefile is not a shell script, different syntax.
Unpack attached scripts to cmix\src\, update gcc path in g.bat (C:\MinGW510\bin to your path with g++.exe), then run g.bat.

[Mauro Vezzosi]

Cmix commit 2019/12/05, changing from layer_norm to rms_norm: yes, rms_norm looks better than layer_norm.
How much does cmix (or lstm-compress) improve?

[byronknoll]

Cmix commit 2019/12/05, changing from layer_norm to rms_norm: yes, rms_norm looks better than layer_norm.
How much does cmix (or lstm-compress) improve?

cmix on enwik8: 14838332 -> 14834133
cmix on enwik9: 115714367 -> 115638939

[encode]

Results on my PC:
CPU: Intel Core i5-9600K @ 4.8 GHz
MoBo: MSI Z390 GAMING EDGE
RAM: HyperX Predator 32 GB (2x16) 3200 MHz CL16
Storage: Samsung 970 EVO Plus 250 GB M.2 NVMe SSD

C:\cmix>timer cmix -c enwik8 enwik8.cmix18

Timer 3.01  Copyright (c) 2002-2003 Igor Pavlov  2003-07-10
100000000 bytes -> 15384933 bytes in 126675.67 s.
cross entropy: 1.231

Kernel Time  =    12.140 = 00:00:12.140 =   0%
User Time    = 126660.828 = 35:11:00.828 =  99%
Process Time = 126672.968 = 35:11:12.968 =  99%
Global Time  = 126675.921 = 35:11:15.921 = 100%

ENWIK9 results will take some time...