Page 1 of 2 12 LastLast
Results 1 to 30 of 50

Thread: Transformation for UIQ2

  1. #1
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts

    Transformation for UIQ2

    uiq2_t.cpp is simple transformation for Generic Compression Benchmark.
    It divides output from UIQ2 into 2 files. First contains well compressible data: three 1 byte values for each Turing machine -> [length of first random part, length of individual repeated fragment, total size of repetitions]. Second file contains almost random data. You can reconstruct original file from this two files.

    Usage:
    To transform: uiq2_t e input output1 output2
    To reconstruct: uiq2_t d input1 input2 output

    Results:
    uiq2.exe uiq2
    uiq2_t.exe e uiq2 o1 o2
    zpaq.exe cmax2.cfg o1.zpaq o1

    uiq2 ... 6517767 bytes
    o1.zpaq ... 1479848 bytes
    o2 ... 1336777 bytes
    uiq2.pmm ... 3004861 bytes
    -> score = (1479848+1336777)/3004861=0.9373
    Attached Files Attached Files
    Last edited by Jan Ondrus; 15th September 2009 at 00:16.

  2. #2
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Thanks Jan! Here's my own speed optimised build...
    Attached Files Attached Files

  3. #3
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    Wow! I just tried it, got .9371 in 65s. paq8k3 -9u takes 3 hours.

  4. #4
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    uiq2_t | PAQ8PF b1 (- -> 0.9399, 142 sec, 1165.8 MB. Tested on my AMD Sempron 2400+ machine.

  5. #5
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    uiq_2 | paq8px_V64 -> 0.9299, 705 sec, 1104 MB

  6. #6
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts

    improved version

    uiq2_t2|zpaq cmax2.cfg -> 0.9209
    Attached Files Attached Files
    Last edited by Jan Ondrus; 15th September 2009 at 12:16.

  7. #7
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    962
    Thanks
    573
    Thanked 397 Times in 295 Posts
    Hi!
    when I've been trying to use both uiq_t or uiq_t2 versions I've got message that library libgcc_s_dw2-1.dll is missing.
    LovePimple version put message: ERROR!ERROR!ERROR!ERROR!ERROR!ERROR!ERROR! probably due to the same problem.

    Darek

  8. #8
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    You may get ERROR!ERROR!ERROR!ERROR!ERROR!ERROR!ERROR! message because of wrong input file. Input file needs to be output from uiq2.exe program (http://mattmahoney.net/dc/uiq/uiq2.zip)

    Quote Originally Posted by Darek View Post
    Hi!
    when I've been trying to use both uiq_t or uiq_t2 versions I've got message that library libgcc_s_dw2-1.dll is missing.
    Replaced exe with one compiled with -static-libgcc switch.
    Last edited by Jan Ondrus; 15th September 2009 at 12:22.

  9. #9
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    962
    Thanks
    573
    Thanked 397 Times in 295 Posts
    Thank you for information. It's my misunderstanding.
    However, have you planned to made such transformation algorithm for any file? It could be used for all low redundant files.
    Darek

  10. #10
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    Quote Originally Posted by Darek View Post
    However, have you planned to made such transformation algorithm for any file? It could be used for all low redundant files.
    Darek
    It is not possible. This trasformation is useful and optimised only for data from Generic Compression Benchmark. It is specific transformation which can't be used for general data.

  11. #11
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    962
    Thanks
    573
    Thanked 397 Times in 295 Posts
    Hi!
    my scores for:

    1) uiq2_t|paq8k3 -8u 0.9391
    2) uiq2_t|paq8k3 -8 0.9317

    However, I've packed both output files from uiq2_t. Second output file is hard to pack, but packed changes score from 0.9320 to 0.9317.

    Darek

  12. #12
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    962
    Thanks
    573
    Thanked 397 Times in 295 Posts
    I've tested uiq2_t2 transformation, and score for uiq2_t2|paq8k3 -8 = 0.9211
    Darek

  13. #13
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    Quote Originally Posted by Darek View Post
    I've tested uiq2_t2 transformation, and score for uiq2_t2|paq8k3 -8 = 0.9211
    Darek
    It is still worse than:
    Code:
    uiq2_t2|zpaq cmax2.cfg -> 0.9209
    uiq2.exe uiq2
    uiq2_t2.exe uiq2 o1 o2
    zpaq.exe cmax2.cfg o1.zpaq o1
    o1.zpaq=1397233 bytes, o2=1371827 bytes,uiq2.pmm=3006784 bytes
    -> score=(1397233+1371827)/3006784=0.9209

  14. #14
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    uiq2_t2|PAQ8PF b1 (- -> 0.9220, 172 sec, 1165.8 MB. Tested on my AMD Sempron 2400+ machine.

  15. #15
    Member
    Join Date
    Aug 2009
    Location
    Bari
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    My test:

    uiq2.exe uiq2
    uiq2_t2.exe e uiq2 o1 o2
    zpaq.exe cmax2.cfg o1.zpaq o1
    pause

    (1.397.180+1.372.45/3.006.531 = 0,9212

    uiq2.exe uiq2
    uiq2_t2.exe e uiq2 o1 o2
    paq8pf.exe -8 o1
    pause

    (1.400.361+1.372.554)/3.009.180 = 0,9214
    Last edited by PiPPoNe92; 15th September 2009 at 19:06.

  16. #16
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    Quote Originally Posted by PiPPoNe92 View Post
    My test:



    (1396874+137008/3006784 = 0,9202

    Is it right "3006784", how I measure the file size of "uiq2.pmm"??
    No. You need PPMonstr from here http://compression.ru/ds/ppmdj1.rar.
    Then use command:
    Code:
    ppmonstr e uiq
    Everything is decribed in Rules section here: http://mattmahoney.net/dc/uiq/
    Last edited by Jan Ondrus; 16th September 2009 at 10:05.

  17. #17
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by PiPPoNe92 View Post
    Is it right "3006784", how I measure the file size of "uiq2.pmm"??
    Please read the following:

    Quote Originally Posted by Matt Mahoney
    To submit a result:

    * Download uiq2 and unzip the file. It contains a source code file (uiq2.cpp, licensed under GPL), and a Windows executable, uiq2.exe.
    * Run as follows: uiq2 filename to create filename (the name doesn't matter).
    * Compress the file (about 6.5 MB) with the compressor you are testing.
    * Compress the same file as follows: ppmonstr e filename
    * Divide the size of the output of the test compressor by the size of filename.pmm (about 3 MB).
    * Truncate the result to 4 decimal places (round down, e.g. 1.56789 rounds to 1.567.

  18. #18
    Member
    Join Date
    Aug 2009
    Location
    Bari
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    yes, done.

  19. #19
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Its better to compress both o1 and o2 files with paq8pf.

    paq8pf.exe -8 uiq o1 o2

  20. #20
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    Retested uiq2_t2 + zpaq with max.cfg and max2.cfg:
    Code:
    max.cfg
    x1:0.92111788
    x2:0.92119469
    x3:0.92136144
    average score=0.9212
    
    max2.cfg
    y1:0.92183323
    y2:0.92187385
    y3:0.92099999
    average score=0.9215

  21. #21
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts

    New record: 0.8818

    I managed to further improve transformation so it produces smaller output2 file. Last bit for each Turing machine was always 1 -> it does not need to be saved in output2 file.
    Quick test uiq2_t3 + zpaq cmax:
    output1.zpaq=1397344 bytes
    output2=1252489 bytes
    uiq2.pmm=3004767 bytes
    -> score=0.8818

    EDIT:
    output2.zpaq=1250605 bytes
    -> score=0.8812
    Attached Files Attached Files
    Last edited by Jan Ondrus; 17th September 2009 at 11:39.

  22. #22
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    962
    Thanks
    573
    Thanked 397 Times in 295 Posts
    My test uiq2_t3 + paq8k3:
    output1.paq8k3=1398017 bytes
    output2.pak8k3=1251125 bytes
    output summary = 2649142 bytes
    uiq2.pmm=3006024 bytes
    -> score=0.8813
    Darek

  23. #23
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    And, very close on their heels...

    uiq2_t3|paq8pf b2 (-:

    uiq2.pmm = 3,005,757 bytes
    output1.paq8pf = 1,400,213 bytes
    output2.pak8pf = 1,251,439 bytes
    1400213 + 1251439 = 2,651,652 bytes


    2651652/3005757 = 0.8821

    Memory used by paq8pf for compression = 1165.8 MB

  24. #24
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts

    new record: 0.8773

    uiq2_t4:
    - last but one bit is sometimes (if repeated pattern found) more often =0 (66% is 0, 33% is 1) so it is saved in output1 file instead of output2 now.
    - thanks to schnaader for his ideas in this thread: http://encode.su/forum/showthread.php?t=464

    uiq2_t4+zpaq cmax.cfg:
    Code:
    output1.zpaq=1452850 bytes
    output2.zpaq=1183690 bytes
    uiq2.pmm=3005208 bytes
    -> score=0.8773
    Attached Files Attached Files
    Last edited by Jan Ondrus; 17th September 2009 at 12:44.

  25. #25
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    962
    Thanks
    573
    Thanked 397 Times in 295 Posts
    My score for uiq2_t4 and my favourite paq8k3 at option -8 are:

    output1.paq8k3=1453321 bytes
    output2.pak8k3=1183298 bytes
    output summary = 2636619 bytes

    uiq2.pmm=3006024 bytes

    -> score=0.8771

    Darek

  26. #26
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    962
    Thanks
    573
    Thanked 397 Times in 295 Posts
    Matt,
    I think that to be fair my scores of GCB like this above should be noticed as Jan Ondrus on GCB page. He is uiq2_t4 builder....

    Darek

  27. #27
    Expert
    Matt Mahoney's Avatar
    Join Date
    May 2008
    Location
    Melbourne, Florida, USA
    Posts
    3,255
    Thanks
    306
    Thanked 779 Times in 486 Posts
    Yeah, but I did not write most of the programs I tested either.

  28. #28
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    uiq2_t4|paq8pf b2 (-:

    uiq2.pmm = 3,005,757 bytes
    output1.paq8pf = 1,456,741 bytes
    output2.pak8pf = 1,183,633 bytes
    1456741 + 1183633 = 2,640,374 bytes


    2640374/3005757 = 0.8784

    Memory used by paq8pf for compression = 1165.8 MB

  29. #29
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    I got an error message from uiq2_t4 : "ERROR: c>255: 1"...
    Does this ring any bell ?

    Code:
    \uiq2_t4>uiq2 1
    Creating random seed...
    3267/2048 deltas, 1619/1536 locations, 1543/1536 intervals in 12.415 s
    to re-create: uiq2 1 VZOOQUZVDDXLAJPDKTSOBISDVNEW 1000000
    292758 programs halted, 633096 timed out, 74146 output premature NUL
    Created file 1
    
    \uiq2_t4>uiq2_t4 e 1 o1 o2
    ERROR: c>255: 1
    \uiq2_t4>

  30. #30
    Programmer Jan Ondrus's Avatar
    Join Date
    Sep 2008
    Location
    Rychnov nad Kněžnou, Czech Republic
    Posts
    278
    Thanks
    33
    Thanked 137 Times in 49 Posts
    Quote Originally Posted by pat357 View Post
    I got an error message from uiq2_t4 : "ERROR: c>255: 1"...
    Fixed this problem in uiq2_t4b. No other changes.

    Quote Originally Posted by Matt Mahoney View Post
    0.8773 uiq2_t4|ppmonstr (PPM) Sep 17 2009 Jan Ondrus
    You listed wrong program in Generic Compression Benchmark. I tested uiq2_t4|zpaq cmax.cfg.
    Attached Files Attached Files

Page 1 of 2 12 LastLast

Similar Threads

  1. UIQ2 statistics
    By schnaader in forum Data Compression
    Replies: 5
    Last Post: 17th September 2009, 22:36
  2. Delta transformation
    By encode in forum Forum Archive
    Replies: 16
    Last Post: 4th January 2008, 12:13

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •