Transformation for UIQ2

[Jan Ondrus]

uiq2_t.cpp is simple transformation for Generic Compression Benchmark.
It divides output from UIQ2 into 2 files. First contains well compressible data: three 1 byte values for each Turing machine -> [length of first random part, length of individual repeated fragment, total size of repetitions]. Second file contains almost random data. You can reconstruct original file from this two files.

Usage:
To transform: uiq2_t e input output1 output2
To reconstruct: uiq2_t d input1 input2 output

Results:
uiq2.exe uiq2
uiq2_t.exe e uiq2 o1 o2
zpaq.exe cmax2.cfg o1.zpaq o1

uiq2 ... 6517767 bytes
o1.zpaq ... 1479848 bytes
o2 ... 1336777 bytes
uiq2.pmm ... 3004861 bytes
-> score = (1479848+1336777)/3004861=0.9373

[LovePimple]

Thanks Jan! Here's my own speed optimised build...

[Matt Mahoney]

Wow! I just tried it, got .9371 in 65s. paq8k3 -9u takes 3 hours.

[LovePimple]

uiq2_t | PAQ8PF b1 (- -> 0.9399, 142 sec, 1165.8 MB. Tested on my AMD Sempron 2400+ machine.

[Matt Mahoney]

uiq_2 | paq8px_V64 -> 0.9299, 705 sec, 1104 MB

[Jan Ondrus]

uiq2_t2|zpaq cmax2.cfg -> 0.9209

[Darek]

Hi!
when I've been trying to use both uiq_t or uiq_t2 versions I've got message that library libgcc_s_dw2-1.dll is missing.
LovePimple version put message: ERROR!ERROR!ERROR!ERROR!ERROR!ERROR!ERROR! probably due to the same problem.

Darek

[Jan Ondrus]

You may get ERROR!ERROR!ERROR!ERROR!ERROR!ERROR!ERROR! message because of wrong input file. Input file needs to be output from uiq2.exe program (http://mattmahoney.net/dc/uiq/uiq2.zip)

Hi!
when I've been trying to use both uiq_t or uiq_t2 versions I've got message that library libgcc_s_dw2-1.dll is missing.

Replaced exe with one compiled with -static-libgcc switch.

[Darek]

Thank you for information. It's my misunderstanding.
However, have you planned to made such transformation algorithm for any file? It could be used for all low redundant files.
Darek

[Jan Ondrus]

However, have you planned to made such transformation algorithm for any file? It could be used for all low redundant files.
Darek

It is not possible. This trasformation is useful and optimised only for data from Generic Compression Benchmark. It is specific transformation which can't be used for general data.

[Darek]

Hi!
my scores for:

1) uiq2_t|paq8k3 -8u 0.9391
2) uiq2_t|paq8k3 -8 0.9317

However, I've packed both output files from uiq2_t. Second output file is hard to pack, but packed changes score from 0.9320 to 0.9317.

Darek

[Darek]

I've tested uiq2_t2 transformation, and score for uiq2_t2|paq8k3 -8 = 0.9211
Darek

[Jan Ondrus]

I've tested uiq2_t2 transformation, and score for uiq2_t2|paq8k3 -8 = 0.9211
Darek

It is still worse than:

uiq2_t2|zpaq cmax2.cfg -> 0.9209

uiq2.exe uiq2
uiq2_t2.exe uiq2 o1 o2
zpaq.exe cmax2.cfg o1.zpaq o1
o1.zpaq=1397233 bytes, o2=1371827 bytes,uiq2.pmm=3006784 bytes
-> score=(1397233+1371827)/3006784=0.9209

[LovePimple]

uiq2_t2|PAQ8PF b1 (- -> 0.9220, 172 sec, 1165.8 MB. Tested on my AMD Sempron 2400+ machine.

[PiPPoNe92]

My test:

uiq2.exe uiq2
uiq2_t2.exe e uiq2 o1 o2
zpaq.exe cmax2.cfg o1.zpaq o1
pause

(1.397.180+1.372.45/3.006.531 = 0,9212

uiq2.exe uiq2
uiq2_t2.exe e uiq2 o1 o2
paq8pf.exe -8 o1
pause

(1.400.361+1.372.554)/3.009.180 = 0,9214

[Jan Ondrus]

My test:



(1396874+137008/3006784 = 0,9202

Is it right "3006784", how I measure the file size of "uiq2.pmm"??

No. You need PPMonstr from here http://compression.ru/ds/ppmdj1.rar.
Then use command:

ppmonstr e uiq

Everything is decribed in Rules section here: http://mattmahoney.net/dc/uiq/

[LovePimple]

Is it right "3006784", how I measure the file size of "uiq2.pmm"??

Please read the following:

To submit a result:

* Download uiq2 and unzip the file. It contains a source code file (uiq2.cpp, licensed under GPL), and a Windows executable, uiq2.exe.
* Run as follows: uiq2 filename to create filename (the name doesn't matter).
* Compress the file (about 6.5 MB) with the compressor you are testing.
* Compress the same file as follows: ppmonstr e filename
* Divide the size of the output of the test compressor by the size of filename.pmm (about 3 MB).
* Truncate the result to 4 decimal places (round down, e.g. 1.56789 rounds to 1.567.

[PiPPoNe92]

yes, done.

[LovePimple]

Its better to compress both o1 and o2 files with paq8pf.

paq8pf.exe -8 uiq o1 o2

[Jan Ondrus]

Retested uiq2_t2 + zpaq with max.cfg and max2.cfg:

max.cfg
x1:0.92111788
x2:0.92119469
x3:0.92136144
average score=0.9212

max2.cfg
y1:0.92183323
y2:0.92187385
y3:0.92099999
average score=0.9215

[Jan Ondrus]

I managed to further improve transformation so it produces smaller output2 file. Last bit for each Turing machine was always 1 -> it does not need to be saved in output2 file.
Quick test uiq2_t3 + zpaq cmax:
output1.zpaq=1397344 bytes
output2=1252489 bytes
uiq2.pmm=3004767 bytes
-> score=0.8818

EDIT:
output2.zpaq=1250605 bytes
-> score=0.8812

[Darek]

My test uiq2_t3 + paq8k3:
output1.paq8k3=1398017 bytes
output2.pak8k3=1251125 bytes
output summary = 2649142 bytes
uiq2.pmm=3006024 bytes
-> score=0.8813
Darek

[LovePimple]

And, very close on their heels...

uiq2_t3|paq8pf b2 (-:

uiq2.pmm = 3,005,757 bytes
output1.paq8pf = 1,400,213 bytes
output2.pak8pf = 1,251,439 bytes
1400213 + 1251439 = 2,651,652 bytes


2651652/3005757 = 0.8821

Memory used by paq8pf for compression = 1165.8 MB

[Jan Ondrus]

uiq2_t4:
- last but one bit is sometimes (if repeated pattern found) more often =0 (66% is 0, 33% is 1) so it is saved in output1 file instead of output2 now.
- thanks to schnaader for his ideas in this thread: http://encode.su/forum/showthread.php?t=464

uiq2_t4+zpaq cmax.cfg:

output1.zpaq=1452850 bytes
output2.zpaq=1183690 bytes
uiq2.pmm=3005208 bytes
-> score=0.8773

[Darek]

My score for uiq2_t4 and my favourite paq8k3 at option -8 are:

output1.paq8k3=1453321 bytes
output2.pak8k3=1183298 bytes
output summary = 2636619 bytes

uiq2.pmm=3006024 bytes

-> score=0.8771

Darek

[Darek]

Matt,
I think that to be fair my scores of GCB like this above should be noticed as Jan Ondrus on GCB page. He is uiq2_t4 builder....

Darek

[Matt Mahoney]

Yeah, but I did not write most of the programs I tested either.

[LovePimple]

uiq2_t4|paq8pf b2 (-:

uiq2.pmm = 3,005,757 bytes
output1.paq8pf = 1,456,741 bytes
output2.pak8pf = 1,183,633 bytes
1456741 + 1183633 = 2,640,374 bytes


2640374/3005757 = 0.8784

Memory used by paq8pf for compression = 1165.8 MB

[pat357]

I got an error message from uiq2_t4 : "ERROR: c>255: 1"...
Does this ring any bell ?

\uiq2_t4>uiq2 1
Creating random seed...
3267/2048 deltas, 1619/1536 locations, 1543/1536 intervals in 12.415 s
to re-create: uiq2 1 VZOOQUZVDDXLAJPDKTSOBISDVNEW 1000000
292758 programs halted, 633096 timed out, 74146 output premature NUL
Created file 1

\uiq2_t4>uiq2_t4 e 1 o1 o2
ERROR: c>255: 1
\uiq2_t4>

[Jan Ondrus]

I got an error message from uiq2_t4 : "ERROR: c>255: 1"...

Fixed this problem in uiq2_t4b. No other changes.

0.8773 uiq2_t4|ppmonstr (PPM) Sep 17 2009 Jan Ondrus

You listed wrong program in Generic Compression Benchmark. I tested uiq2_t4|zpaq cmax.cfg.