Today I've made a new C version, this time I've fixed bugs in command line parsing and added prefetching that actually works, but only on encoding.
Compile with -std=c99 -O3 -march=native -DGCC_PREFETCH and preferably for 64-bit architecture.
Encoding without prefetching:
Code:
piotrek@p5q-pro:~/NetBeansProjects/TarsaLZP$ time dist/Release/GNU-Linux-x86/tarsalzp encode fi=/home/piotrek/datacomp/enwik/enwik8 fo=/home/piotrek/Pulpit/clzp lzpLowMaskSize=25 lzpHighMaskSize=30
TarsaLZP
Author: Piotr Tarsa
Completed!
real 0m17.219s
user 0m15.905s
sys 0m1.192s
Encoding with prefetching:
Code:
piotrek@p5q-pro:~/NetBeansProjects/TarsaLZP$ time dist/ReleaseP/GNU-Linux-x86/tarsalzp encode fi=/home/piotrek/datacomp/enwik/enwik8 fo=/home/piotrek/Pulpit/clzp lzpLowMaskSize=25 lzpHighMaskSize=30
TarsaLZP
Author: Piotr Tarsa
Completed!
real 0m12.259s
user 0m10.853s
sys 0m1.220s
Decoding - without prefetching, as prefetching didn't work in that case:
Code:
piotrek@p5q-pro:~/NetBeansProjects/TarsaLZP$ time dist/ReleaseP/GNU-Linux-x86/tarsalzp decode fi=/home/piotrek/Pulpit/clzp fo=/home/piotrek/Pulpit/cdec
TarsaLZP
Author: Piotr Tarsa
Completed!
real 0m18.830s
user 0m17.397s
sys 0m1.252s
As to options, program should tell about available options and their default values. If you're curious what are valid options there's validation function:
Code:
bool optionsValid() {
return lzpLowContextLength > ppmOrder
&& lzpLowContextLength <= lzpHighContextLength
&& lzpHighContextLength <= 8
&& lzpLowMaskSize >= 15
&& lzpLowMaskSize <= 30
&& lzpHighMaskSize >= 15
&& lzpHighMaskSize <= 30
&& ppmOrder >= 1
&& ppmOrder <= 2
&& ppmInit >= 1
&& ppmInit <= 127
&& ppmStep >= 1
&& ppmStep <= 127
&& ppmLimit >= ppmInit * 256
&& ppmLimit <= 32767 - ppmStep;
}
Guidance for setting options:
In general increasing *MaskSize always helps, but memory usage grows exponentially with that parameters. For example if you set lzpHighMaskSize to 30, then high LZP model will consume 2^30 uint16_t's, ie 2 GiB of memory. Context lengths are tuned for enwik8/ enwik9, probably for less compressible files lower values would be better. ppmOrder on small files (far below one megabyte) probably should be set to 1. For incompressible files set ppmInit to some relatively high value, like 60 and set ppmStep to something lower, like 20.
If you set lzpLowMaskSize == lzpHighMaskSize and lzpLowContextLength == lzpHighContextLength then effectively there will be one LZP model and program should work faster.