fpaqvm (vm/jit)

[kaitz]

This compressor is based on fpaq0. It uses virtual machine to execute predictor. There are only linux executable files (x86), no soruce yet. fpaqvmnojit and fpaqvmjit. First one will execute user provided config file in virtual machine. File is translated into bytecode and then executed in vm. Second executable file translates input config file into bytecode and then emits x86 opcode and executes it.


There are two config files. test1.cfg uses fpaq0 model. test2.cfg uses fpaqc model. test1.cfg is identical to original compressor. Second one is not. There are some differences in encoder etc.

Config file language is subset of c.

$ ./fpaq0 c calgary.tar calgary.tar.fpaq0
calgary.tar (3152896 bytes) -> calgary.tar.fpaq0 (1903024 bytes) in 1.02 s.
$ ./fpaqvmjit c test1.cfg calgary.tar calgary.tar.fpaqvmjit
calgary.tar (3152896 bytes) -> calgary.tar (1903024 bytes) in 2.74 s.
$ ./fpaqvmjit c test2.cfg calgary.tar calgary.tar.fpaqvmjit
calgary.tar (3152896 bytes) -> calgary.tar (1682796 bytes) in 1.79 s.
$ ./fpaqvmnojit c test1.cfg calgary.tar calgary.tar.fpaqvmnojit
calgary.tar (3152896 bytes) -> calgary.tar (1903024 bytes) in 53.29 s.
$ ./fpaqvmnojit c test2.cfg calgary.tar calgary.tar.fpaqvmnojit
calgary.tar (3152896 bytes) -> calgary.tar (1682796 bytes) in 35.59 s.

[Matt Mahoney]

Did you write your own C compiler? It seems like there is a lot of legal code (checked with gcc -c -Wall) that results in no errors but a size 0 output, like declaring a variable at the beginning of update() with an initial value like:

int B=512; // fails
int B; // OK

It also fails if I try to change t from int* to an array, or to unsigned short*. Still, a very interesting idea to put a subset of C source code to decompress into a self describing archive. I was able to modify test2.cfg from order 0 to order 1 to improve compression, just as a quick test.

[kaitz]

I did not write compiler myself. I'm using https://github.com/EarlGray/c4/ https://github.com/rswier/c4/ smalest i could find.
Added for statement and removed some parts. As they are GPL i think i can use them.
Arrays must be allocated before use with malloc, global variables can be used like:
int a; //global
int *b;
void main(){
int myzise;
a=2; // in first usage
myzise=a*10;
b=malloc(myzise*sizeof(int));
a=b[5];
}
int and char are only supported. Local variables must be at the start of function.

I just wanted to see if its possible.

[kaitz]

Source code

$ ./fpaqvmjit c test1.cfg  enwik8 enwik8.0
enwik8 (100000000 bytes) -> enwik8 (63391013 bytes) in 89.43 s.
$ ./fpaqvmjit c test2.cfg  enwik8 enwik8.1
enwik8 (100000000 bytes) -> enwik8 (61280454 bytes) in 59.27 s.
$ ./fpaqvmjit c test3.cfg  enwik8 enwik8.2
enwik8 (100000000 bytes) -> enwik8 (45979831 bytes) in 67.38 s.

[kaitz]

test4.cfg o2

$ ./fpaqvmjit c test4.cfg  enwik8 enwik8.3
enwik8 (100000000 bytes) -> enwik8 (36950806 bytes) in 75.33 s.

test3.cfg is o1 not o2.