bigm file compressor

[suryakandau@yahoo.co.id]

may someone on this forum run it for enwik8 and enwik9 or darek benchmark ?? this is use neural network so the running time can so long.
thanx you very much

[Shelwien]

https://www.virustotal.com/gui/file/...d632/detection
Seems like renamed cmix?

[suryakandau@yahoo.co.id]

https://www.virustotal.com/gui/file/...d632/detection
Seems like renamed cmix?

No.

[Shelwien]

What "no"? I looked at the debuginfo in it, and its got all the cmix classes

 0001:00000500       Manager::Manager(void)
 0001:0000088E       Manager::UpdateHistory(void)
 0001:000008D0       Manager::UpdateWords(void)
 0001:00000A36       Manager::UpdateRecentBytes(void)
 0001:00000A68       Manager::Perceive(int)
 0001:00000B9C       Manager::AddContext(std::unique_ptr<Context,std::default_delete<Context>>)
 0001:00000C36       Manager::AddBitContext(std::unique_ptr<BitContext,std::default_delete<BitContext>>)
 0001:00000C90       Predictor::GetNumNeurons(void)
 0001:00000D46       Predictor::Predict(void)
 0001:00000FE4       Predictor::Perceive(int)
 0001:000011D0       Predictor::Add(Model *)
 0001:00001290       Predictor::AddDMC(void)
 0001:0000139C       Predictor::AddByteRun(void)
 0001:000019C8       Predictor::AddRunMap(void)
 0001:00002004       Predictor::AddNonstationary(void)
 0001:00002728       Predictor::AddMatch(void)
 0001:00002FA8       Predictor::AddDoubleIndirect(void)
 0001:0000386A       Predictor::AddDirect(void)
 0001:00003F1C       Predictor::AddSparse(void)
 0001:000051B2       Predictor::AddEnglish(void)
 0001:000065C4       Predictor::AddByteModel(ByteModel *)
 0001:00006686       Predictor::AddPPM(void)
 0001:000067A6       Predictor::Add(int,Mixer *)
 0001:000068CC       Predictor::AddPAQ8L(void)
 0001:00006A16       Predictor::AddPAQ8HP(void)
 0001:00006B62       Predictor::AddPAQ8H2(void)
 0001:00006CAC       Predictor::AddSSE(void)
 0001:000073BE       Predictor::AddMixers(void)
 0001:0000A3D4       Predictor::Predictor(void)

https://github.com/byronknoll/cmix/b...-manager.h#L13
https://github.com/byronknoll/cmix/b...redictor.h#L26

[suryakandau@yahoo.co.id]

What "no"? I looked at the debuginfo in it, and its got all the cmix classes

 0001:00000500       Manager::Manager(void)
 0001:0000088E       Manager::UpdateHistory(void)
 0001:000008D0       Manager::UpdateWords(void)
 0001:00000A36       Manager::UpdateRecentBytes(void)
 0001:00000A68       Manager::Perceive(int)
 0001:00000B9C       Manager::AddContext(std::unique_ptr<Context,std::default_delete<Context>>)
 0001:00000C36       Manager::AddBitContext(std::unique_ptr<BitContext,std::default_delete<BitContext>>)
 0001:00000C90       Predictor::GetNumNeurons(void)
 0001:00000D46       Predictor::Predict(void)
 0001:00000FE4       Predictor::Perceive(int)
 0001:000011D0       Predictor::Add(Model *)
 0001:00001290       Predictor::AddDMC(void)
 0001:0000139C       Predictor::AddByteRun(void)
 0001:000019C8       Predictor::AddRunMap(void)
 0001:00002004       Predictor::AddNonstationary(void)
 0001:00002728       Predictor::AddMatch(void)
 0001:00002FA8       Predictor::AddDoubleIndirect(void)
 0001:0000386A       Predictor::AddDirect(void)
 0001:00003F1C       Predictor::AddSparse(void)
 0001:000051B2       Predictor::AddEnglish(void)
 0001:000065C4       Predictor::AddByteModel(ByteModel *)
 0001:00006686       Predictor::AddPPM(void)
 0001:000067A6       Predictor::Add(int,Mixer *)
 0001:000068CC       Predictor::AddPAQ8L(void)
 0001:00006A16       Predictor::AddPAQ8HP(void)
 0001:00006B62       Predictor::AddPAQ8H2(void)
 0001:00006CAC       Predictor::AddSSE(void)
 0001:000073BE       Predictor::AddMixers(void)
 0001:0000A3D4       Predictor::Predictor(void)

https://github.com/byronknoll/cmix/b...-manager.h#L13
https://github.com/byronknoll/cmix/b...redictor.h#L26

Just a name but the content is different. You can check the SHA.is it has the same sha value or not

[Bulat Ziganshin]

lzturbo reincarnation

[JamesB]

Just a name but the content is different. You can check the SHA.is it has the same sha value or not

It's possible it was derived from cmix but is still substantially different, but the evidence of all those names is that it needs to be open source given cmix is GPL.

Can we please get a copy of the source-code therefore so people can judge how it compares, and also compile it up without the potential virus.

[Shelwien]

There's no virus, so it can be tested and compared with cmix results.
I don't think its the most recent cmix version.

[Bulat Ziganshin]

scanning for known viruses can't detect all trojan horses, f.e. program deleting all disk files

[Shelwien]

I actually looked at decompiler output to check that, just found cmix instead.
Of course I can't give 100% guarantee, but there's nothing obvious, and I think that
the person able to hide a trojan from me would be at least able to build cmix with -s after replacing the name.

[suryakandau@yahoo.co.id]

I actually looked at decompiler output to check that, just found cmix instead.
Of course I can't give 100% guarantee, but there's nothing obvious, and I think that
the person able to hide a trojan from me would be at least able to build cmix with -s after replacing the name.

this is bigm_suryak v2:
improved word model by new hash function
this file contain source and binary too.
please Darek or someone test it and give the result. it uses <=32Gb memory usage

[Gotty]

suryakandau@yahoo.co.id,
Someone may help you out here (not me - I have only 8 gigs), but you are asking for significant resources.
Please consider going shopping for some RAM so that you can try and verify your ideas. This is essential if you are seriously thinking about improving cmix. It will not go easily if you need to wait for someone for days or weeks.

Most importantly, please respect the work of others especially if you are planning standing on their shoulders.

[Shelwien]

I actually posted this specifically for him: https://encode.su/threads/3242-googl...ession-testing

Also he really did some context tweaking in paq8pxd... presuming that this is the same thing, I'm kinda okay with renaming,
since its better than having a paq8pxd47_bwtXX branch. Just have to be clear with authorship.

[kaitz]

It has 3 new or modified files, i assume in cmix version x from 2015.
2 new paq8(l,hp) models with reduced memory. Hash functions and 9 context in wordmodel.
I think it will be slow. To bad there isin't SIMD code in mixers. Still it has some SIMD in it, probably auto from compiler. Checked only paq8l.o.

[suryakandau@yahoo.co.id]

I actually posted this specifically for him: https://encode.su/threads/3242-googl...ession-testing

Also he really did some context tweaking in paq8pxd... presuming that this is the same thing, I'm kinda okay with renaming,
since its better than having a paq8pxd47_bwtXX branch. Just have to be clear with authorship.

Paq8pxd branch from pa8px69 and then start with new thread.

[suryakandau@yahoo.co.id]

bigm_suryak v3
enwik8 17,858,121 bytes
​compression time 132776.36 s

[suryakandau@yahoo.co.id]

bigm_suryak v3 using level 6. i do not try with level 9 or 10 because my laptop has only 16Gb.
bigm_suryak v4 on progress

[suryakandau@yahoo.co.id]

bigm_suryak v3
enwik8 17,858,121 bytes
​compression time 132776.36 s

It only takes ~1.3 Gb memory

[suryakandau@yahoo.co.id]

It only takes ~1.3 Gb memory

XML.tar from Silesia benchmark without Precomp and memory only ~1.3 Gb the result is 265136 byte

[suryakandau@yahoo.co.id]

XML.tar from Silesia benchmark without Precomp and memory only ~1.3 Gb the result is 265136 byte

bigm_suryak v8
- improve word model
-use only 1.3 gb memory
xml file from silesia benchmark
without precomp: 264725 bytes

[byronknoll]

@suryakandau, please distribute the source code of bigm along with the releases. Since cmix code is GPL, and bigm contains cmix code, bigm also needs to be open source.

[suryakandau@yahoo.co.id]

bigm_suryak v8
- improve word model
-use only 1.3 gb memory
xml file from silesia benchmark
without precomp: 264725 bytes

enwik8 17809514 bytes using ~1.3 gb memory
​

[suryakandau@yahoo.co.id]

bigm_suryak v9
improve wordmodel
xml file from silesia benchmark 264409 bytes using ~1.3 gb memory
this is archive file contain source code and binary

[suryakandau@yahoo.co.id]

bigm_suryak v9
improve wordmodel
xml file from silesia benchmark 264409 bytes using ~1.3 gb memory
this is archive file contain source code and binary

Enwik8 17801098 bytes

[suryakandau@yahoo.co.id]

@suryakandau, please distribute the source code of bigm along with the releases. Since cmix code is GPL, and bigm contains cmix code, bigm also needs to be open source.

@byron may you run bigm using paq8hp(11) paq8h2(11) paq8l(10) on enwik8 ? I just want to know how much enwik8 can be compressed ...thank you

[suryakandau@yahoo.co.id]

bigm_suryak v9.1
xml 264149 bytes

[suryakandau@yahoo.co.id]

bigm_suryak v9.2
- improve word model
- improve nest model

xml (silesia benchmark) 263672 bytes
astramina.fna 361269 bytes
enwik8 on progress..



​

[suryakandau@yahoo.co.id]

bigm_suryak v9.2
- improve word model
- improve nest model

xml (silesia benchmark) 263672 bytes
astramina.fna 361269 bytes
enwik8 on progress..



​

enwik8 17661844 bytes using only ~1.3 gb memory
enwik9 on progress...maybe the result ~13x.xxx.xxx bytes

this is the source code of bigm_suryak v9.2

[suryakandau@yahoo.co.id]

enwik8 17661844 bytes using only ~1.3 gb memory
enwik9 on progress...maybe the result ~13x.xxx.xxx bytes

this is the source code of bigm_suryak v9.2

Bigm_suryak v9.4

XML from Silesia benchmark 263571 bytes without using precomp n memory only ~1.3gb

[suryakandau@yahoo.co.id]

Bigm_suryak v9.4

XML from Silesia benchmark 263571 bytes without using precomp n memory only ~1.3gb

Bigm_suryak v9.4.1
xml from silesia benchmark 263522 bytes without using precomp and memory only 1.3gb