Results 1 to 17 of 17

Thread: QUAD 1.05a is here!

  1. #1
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Okay, a new version of QUAD has been released! This version is not about super compression, but fast decompression speed. I think this version is pretty close to the final release which I'm planning to release at Source Forge. However, maybe I'll change something in both directions – compression and/or speed. Anyway, enjoy!

    Link:
    quad-1.05a.zip (55 KB)


  2. #2
    Member
    Join Date
    Jun 2008
    Location
    G
    Posts
    372
    Thanks
    26
    Thanked 22 Times in 15 Posts
    coool thx

  3. #3
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Ilia!

  4. #4
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Thanks!

    EDIT: 13,478,664 bytes in total!
    Details about speeds in next table

  5. #5
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quick test...

    Normal mode:

    A10.jpg > 853,522
    AcroRd32.exe > 1,503,699
    english.dic > 914,252
    FlashMX.pdf > 3,832,072
    FP.log > 713,280
    MSO97.dll > 1,914,337
    ohs.doc > 848,925
    rafale.bmp > 1,037,418
    vcfiu.hlp > 708,539
    world95.txt > 627,165

    Total = 12,953,209 bytes


    Max mode:

    A10.jpg > 853,522
    AcroRd32.exe > 1,498,051
    english.dic > 823,666
    FlashMX.pdf > 3,825,201
    FP.log > 632,630
    MSO97.dll > 1,909,522
    ohs.doc > 843,827
    rafale.bmp > 1,013,743
    vcfiu.hlp > 684,776
    world95.txt > 600,075

    Total = 12,685,013 bytes

  6. #6
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    2 Black_Fox

    By the way, I'm very curious about decompression speed of QUAD v1.02a at your test. Currently I'm still thinking about PPM-1 (order-1) with Full Exclusion. QUAD v1.02 uses PPM-1 with no Full Exclusion trick. However, decompression speed must be about the same.

    2 All

    Note that I also tried to use a pure PPM-2 (order-2) with Full Exclusion – this variant doesn't work, since LZ 'eats' all contexts needed for PPM. For example, PPM cannot be used with baseline LZ77 – since pure LZ destroys all contexts needed for context coding. ROLZ keeps low order context untouched, however, the max order is order-1 or tricky order-2 (which can be described also as an enhanced order-1). Therefore, ROLZ with increased MIN_MATCH_LEN can be useful with high order context coders (Malcolm Taylor implementations).

    Some tech details (assume PPM-2X is my enhanced variant of PPM-1):

    PPM-1 uses 256 KB
    PPM-2X uses 1 MB

    Memory usage ~32 MB + 256 KB or 1 MB – i.e. no difference if at all.

    I will keep Full Exclusion, since this really helps with binary and already compressed files a la A10.jpg. The speed is not really affected; however, for PPM-2X speed impact is much more noticeable than for PPM-1.

    Another question is rescale rate – or speed of model adaptation. With PPM-2X I've found value four is the best, however A10.jpg inflates (~834 KB – anyway, this value is not criminal, I guess). With an order-1 this rescale rate is more important thing, with RR of four we can even slightly compress A10.jpg, however such RR leads poor binary compression.

    So anyways, I will run some additional decompression speed tests to find the most efficient solution.

    However, I'm looking forward for benchmarking results. By the way, what do you think about compression ratio and decompression speed of QUAD? Is it works for you?


  7. #7
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quick speed test...

    Test file: "mintest.tar" - Contains: html, txt, gif, bmp, exe, dll, etc
    Decompressed size: 112 MB (117,768,192 bytes)


    Quad v1.05a

    Compressed size: 21.9 MB (22,989,829 bytes)

    Compression speed: 00:02:41.125

    Decompression speed: 00:01:05.343



    Quad v1.04a

    Compressed size: 19.3 MB (20,274,568 bytes)

    Compression speed: 00:03:22.344

    Decompression speed: 00:02:02.078



    CCM v 1.0.6a2

    Compressed size: 21.2 MB (22,289,356 bytes)

    Compression speed: 00:02:49.250

    Decompression speed: 00:02:44.734



    CCM_extra v1.0.6a2

    Compressed size: 19.4 MB (20,396,131 bytes)

    Compression speed: 00:03:06.109

    Decompression speed: 00:03:09.812

  8. #8
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    QUAD is good! By the way, soon a very first BETA version will be available! You may ask - so what? This one will be OpenSource...

  9. #9
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts

  10. #10
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by encode
    This one will be OpenSource..

  11. #11
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 153 Times in 18 Posts
    You announced multi-threading. Is it implemented in this version? On my side it doesnt seem to work.

    Quote Originally Posted by encode
    By the way, what do you think about compression ratio and decompression speed of QUAD? Is it works for you?
    I think the compromise youve found is a really good one! Congratulations to you! Btw. good move to make it OS!

    Regards, Christian

  12. #12
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Christian
    You announced multi-threading. Is it implemented in this version? On my side it doesnt seem to work.
    The most work with multi-threading are done with my CM. This version of QUAD uses PPM. However, it has some speed improvements over older ROLZ-based versions. Since Im going OpenSource I simplified some parts of code, including I eliminated some experimental MT features of my ROLZ. However, all in process...


  13. #13
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by encode
    By the way, Im very curious about decompression speed of QUAD v1.02a at your test.
    Im off for weekend, will test it ASAP.

  14. #14
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    QUAD 1.02a decompression speed is 2574kB/s on my test

  15. #15
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    A few notes:
    + QUAD 1.07BETA must be faster than 1.05a
    + At your test (a set of single files) QUAD must be slower. Why?
    A brief explanation:
    QUAD has a few models:
    1. order-2 with 10-bit context
    2. order-0 with excluded symbols
    3. order-0
    4. Stationary model for match indexes
    In this explanation I'll omit the last model and will keep all eyes on decoder.

    Principal scheme of PPM:
    If order-2 context exists then
    Decode symbol via order-2 context

    If symbol == Escape Symbol then
    Generate a new counter that based on order-0 counter, but has no symbols that has order-2

    Decode symbol via new counter with excluded symbols

    Add symbol to order-0 counter
    Add escape symbol to order-2 counter
    Add symbol to order-2 counter

    Else
    Add symbol to order-2 counter
    End if
    Else
    Decode symbol via order-0 counter

    Add symbol to order-0 counter
    Add escape symbol to order-2 counter
    Add symbol to order-2 counter
    End if

    The "bottleneck" of this code is the process of generating a new counter with excluded symbols. At start of decoding relatively at each symbol coded we invoke such routine, after some data decoded and model gets more statistics, we use order-0 with excluded symbols more rarely – since symbols in order-2 contexts are already seen. At some point of decoding decoder stars using just order-2. So, if we decompressing many files, each time we start decompressing decoder uses many statistics from order-0 with excluded symbols – as a result noticeable speed impact.

    To be honest, such trick as Full Exclusion is incompatible with speed. However, here I use a Small PPM, and on large enough files the speed impact is not so huge.

    Note that in upcoming QUAD 1.08BETA I will improve some parts of code, including PPM encoder and decoder, since in 1.07BETA, some parts of PPM, to be honest, inefficiently implemented. So, expecting a new speed improvements...


  16. #16
    Member Fallon's Avatar
    Join Date
    May 2008
    Location
    Europe - The Netherlands
    Posts
    158
    Thanks
    14
    Thanked 10 Times in 5 Posts
    Looking forward to the next version.

  17. #17
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    By the way, next QUAD 1.08 (a stable release) has a higher compression! Along with a higher compression/decompression speed, an additional error checking and many additional improvements...

    As a proof, download the 'world95.txt' compressed to 596,582 bytes:
    world95.txt.quad


Similar Threads

  1. Quad 1.05a overview
    By encode in forum Forum Archive
    Replies: 47
    Last Post: 21st February 2007, 21:16

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •