15th February 2020, 01:13
Another testing.
Wanted to see how well context actually work (wordmodel). ContextMap collects info and if threshold is reached context is permanently disabled, stats collection also.
enwik6
i(0)=431965, i(3)=324985, i(24)=488462, i(27)=207541, i(33)=493349, i(34)=157440, i(35)=168725, i(36)=179219, i(37)=555562, i(38)=558076, i(45)=425289, i(58)=399287, i(60)=230295, i(61)=210033,
book1
i(0)=520011, i(3)=491253, i(24)=394110, i(25)=220564, i(26)=4994, i(32)=132856, i(33)=490038, i(34)=76312, i(35)=77780, i(36)=80227, i(37)=463017, i(38)=461574, i(45)=421269, i(58)=405311, i(60)=256524, i(61)=131212,
below bad contexts:
+ is enwik6
- is book1
in book i(26) is still ok sortof.
-+ cm.set(hash(++i,x.spafdo, x.spaces,ccword));
-+ cm.set(hash(++i,x.spaces, (x.words&255), (numbers&255)));
-+ cm.set(hash(++i,h, word1,word2,lastUpper<x.wordlen));
- cm.set(hash(++i,text0&0xffffff));
- cm.set(text0&0xfffff);/// i(26)=4994, book1
+ cm.set(hash(++i,word0,number0, data0,xword0));
- cm.set(hash(++i,word0, cword0,isfword));
-+ cm.set(hash(++i,word0,buf(1), word2,isfword));
-+ cm.set(hash(++i,word0,buf(1), word3));
-+ cm.set(hash(++i,word0,buf(1), word4));
-+ cm.set(hash(++i,word0,buf(1), word5));
-+ cm.set(hash(++i,word0,buf(1), word1,word3));
-+ cm.set(hash(++i,word0,buf(1), word2,word3));
-+ cm.set(hash(++i,nl1-nl2,x.col,buf(1),above));
- cm.set(hash(++i,h, llog(wordGap), mask&0x1FF, ));
+ cm.set(hash(x.col,x.wordlen1,above,above1,x.c4&0xfF)); else cm.set(); //wordlist
-+ cm.set(hash(++i,x.col,above^above1,above2 , ((islink)<<8)|)); //wordlist((istemplate)<<9)|
-+ cm.set(hash((*pWord).Hash, h));
book1 compressed
183314 (pxd v72) 100 sec
183752 (pxd vXX skip if i(x)>2024) 88 sec
183288 (pxd vXX no skip) 99 sec
184490 (px v183) 139 sec