Page 1 of 2 12 LastLast
Results 1 to 30 of 48

Thread: Quad 1.05a overview

  1. #1
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Okay, new things like CCM challenging me. Therefore, I brainstormed and make a few very important improvements. List of new features:
    + Some compression improvement
    + Some speed improvement
    + Special optimization for modern processors such as Pentium 4 and Core Duo. That means, I added a multi-threading support [!]

    Note that recommended system configuration for upcoming version is:
    Pentium 4/Core Duo processor
    256 MB RAM

    However, new version must works on other CPUs. Also note that memory usage will be the same.

    Estimated release date: 23 February – 1 March


  2. #2
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by encode
    + Some compression improvement
    + Some speed improvement
    + Special optimization for modern processors such as Pentium 4 and Core Duo. That means, I added a multi-threading support [!]
    Very promising! Especially the multi-threading support sounds very good since there arent many programs utilizing multi-cored CPUs.

  3. #3
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    We wait for that you astonish to us! Hello Encode!

  4. #4
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by encode
    Note that recommended system configuration for upcoming version is:
    Pentium 4/Core Duo processor
    Does this mean that it will be extremely slow (about 30% slower than P4) on my AMD Sempron machine?

  5. #5
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 153 Times in 18 Posts
    Quote Originally Posted by encode
    Okay, new things like CCM challenging me. Therefore, I brainstormed and make a few very important improvements. List of new features:
    + Some compression improvement
    + Some speed improvement
    + Special optimization for modern processors such as Pentium 4 and Core Duo. That means, I added a multi-threading support [!]
    This sounds very promising! Especially the speed improvement + Multithreading support! What speed are we talking about? 2-3 MiB/s!? Im looking forward to your release!

  6. #6
    Member
    Join Date
    Jun 2008
    Location
    G
    Posts
    372
    Thanks
    26
    Thanked 22 Times in 15 Posts
    gogo

  7. #7
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Well, I can write a huge post about what I've done, but I try to write just a few words.

    First, I completely rewrite the core of the Quad data compression engine, from scratch.

    I've made many speed improvements, so new Quad must works faster on any machine, starting from Pentium Pro (out of the order execution tricks).

    How much faster? It depends on machine. I'm tuning performance for Pentium 4. However, Dual Core gets the most benefits from new optimizations. In addition, don't forget about asymmetric nature of the Quad. For example, new Quad unpacks compressed "fp.log" on my P4 within 1...1,5 seconds – i.e. decompression speed ~20 MB/sec. For example, it's impossible with CCM.

    Anyway, I'm still optimizing, testing, tuning...

    Currently, I don't make a decision about new EXE-filter, other things at almost written.


  8. #8
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    While, I'm playing with CM, I've made a very fast version of CM.

    Before, CM was not added to Quad (ROLZ versions) due to speed problems. And goal of my ROLZ is fast decompression.

    Now, I've made some experiments with CM-based ROLZ. Well, I like it! Fast decompression, high binary compression. However, in can enlarge already compressed files and not so good with text files.

    Also new ROLZ engine has two modes:
    Fast (Default)
    Max


    I'm thinking to use this compression in new Quad. Better if TC will be for high compression only and with Quad I will keep something really practical with small memory footprint and really fast decompression speed.

    Usage examples:

    To compress fp.log with Fast mode, type:
    quad e fp.log fp.quad

    To compress fp.log with Max mode:
    quad ex fp.log fp.quad

    Decompression speed is about the same with all modes, or even with Max mode you can get faster decompression.

    Memory usage is the same for all modes and is equal to 32 MB.

    Of course, this new engine is also uses some extra optimizations and multi-threading.

    P.S.
    Started optimizing the speed and got really new compression engine...

  9. #9
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,565
    Thanks
    220
    Thanked 146 Times in 83 Posts
    We wait for with eagerness to try it!

  10. #10
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by encode
    Usage examples:

    To compress fp.log with Fast mode, type:
    quad e fp.log fp.quad

    To compress fp.log with Max mode:
    quad ex fp.log fp.quad
    I like this idea!

  11. #11
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    I think this engine MUST have mode switching. Look at some numbers:

    Test file:
    reaktor.exe, size: 14,446,592 bytes

    Fast mode:
    compression time: 6 sec
    decompression time: 3 sec
    compressed size: 2,028,418 bytes

    Max mode:
    compression time: 39 sec
    decompression time: 2 sec
    compressed size: 1,970,867 bytes

    Like you see, a small compression gain. However, in some cases Max mode makes the difference. For example:

    Test file:
    fp.log, size: 20,617,071 bytes

    Fast mode:
    compression time: 2 sec
    decompression time: 1 sec
    compressed size: 736,033 bytes

    Max mode:
    compression time: 48 sec
    decompression time: less than 1 sec
    compressed size: 628,593 bytes

    So, if you have some time for better compression, use max mode.

    I like LZ!

  12. #12
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by encode
    I think this engine MUST have mode switching. Look at some numbers:
    I agree! Mode switching is a great idea!

  13. #13
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Okay, engine was further improved! Again, look at some results:

    Profile description:
    Fast - Greedy Parsing
    Normal – Enhanced Lazy Evaluation
    Max – Dynamic Programming

    Reaktor.exe (14,446,592 bytes)
    Fast: 2,028,418 bytes [4 sec]
    Normal: 2,003,119 bytes [5 sec]
    Max: 1,970,867 bytes [33 sec]

    TraktorDJStudio3.exe (29,124,024 bytes)
    Fast: 5,442,687 bytes [13 sec]
    Normal: 5,374,309 bytes [15 sec]
    Max: 5,303,577 bytes [92 sec]

    Photoshop.exe (19,533,824 bytes)
    Fast: 6,342,868 bytes [17 sec]
    Normal: 6,272,578 bytes [20 sec]
    Max: 6,198,600 bytes [34 sec]

    fp.log (20,617,071 bytes)
    Fast: 736,033 bytes [1 sec]
    Normal: 719,684 bytes [2 sec]
    Max: 628,593 bytes [38 sec]

    So, I think better Quad will use Normal profile by default.

  14. #14
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by encode
    Profile description:
    Fast - Greedy Parsing
    Normal – Enhanced Lazy Evaluation
    Max – Dynamic Programming
    We will have THREE awesome modes to chose from?

    Quote Originally Posted by encode
    So, I think better Quad will use Normal profile by default.
    Agreed!

  15. #15
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    472
    Thanked 175 Times in 85 Posts
    Hi there

    my benchmarks have been updated.
    http://squeezechart.freehost.ag

    Yours,
    Stpehan

  16. #16
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by encode
    Max mode:
    compression time: 48 sec
    decompression time: less than 1 sec
    Thats so good LZMA, be afraid of Quad!

  17. #17
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Stephan Busch
    my benchmarks have been updated.
    Thank you Stephan!

    Quote Originally Posted by LovePimple
    We will have THREE awesome modes to chose from?
    Currently Quad already has two modes - Normal and Max.

    (usage: quad {e[x]|d} <infile> <outfile>)

    Ive made option parsing like in BBB:

    quad <options> <infile> <outfile>

    ...options should be concatenated...

    so

    quad ex 1.txt 1.quad

    is equal to

    quad xe 1.txt 1.quad

    and even equal to

    quad x 1.txt 1.quad

    to decompress

    quad d 1.quad 1.txt

    ...very useful!

    Also Ive made some experiments with LZP. And what Ive got:

    LZP is very useful with high-order literal coder, but useless with low order ones.

    LZP is slower at decompression compared to ROLZ, but faster at compression.

    ROLZ is not so good with high-order literal coders.

    But the worst thing about new Quad is a poor performance on text files (slightly worse than Quad 1.01a) and it enlarges already compressed files (A10.jpg). However, such issues can be easily fixed with SSE stage, but this trick leads some serious speed impact (2x...2.5x slower decompression). Another solution is adaptive weights, anyway Ill try some... Keeping in mind fast decompression!


  18. #18
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    In addition, I'll try to use improved PPM (v1.03a). Also, this summer (June-August) I'm planning to release Quad as an open source compressor (SourceForge), but probably with more strictly license.


  19. #19
    Tester
    Stephan Busch's Avatar
    Join Date
    May 2008
    Location
    Bremen, Germany
    Posts
    876
    Thanks
    472
    Thanked 175 Times in 85 Posts
    Hi again,

    this seems to be a bit off-topic, but I am a bloody compiling beginner.. could someone please compile chile 0.4 and
    quicklz 1.20 for me?
    And I am still searching for a free webhoster without traffic limit -
    the german providers have too much advertising banners..

    Yours,
    Stephan

  20. #20
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by encode
    Currently Quad already has two modes - Normal and Max.
    Will the release version will have three (Fast, Normal, and Max) as you stated in your previous post?

  21. #21
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Stephan Busch
    compile chile 0.4
    Some time ago I tried to compile this one, but with no luck. My compiler just rejected the code and to compile the source must be modified and lots of time must be spent.

    Quote Originally Posted by Stephan Busch
    quicklz 1.20
    If I will have some spare time, I for sure will try to compile QuickLZ. But, for now, Im fully busy on my job (theatrical premiere is coming) and all my spare time for Quad development.

    Quote Originally Posted by Stephan Busch
    And I am still searching for a free webhoster without traffic limit -
    Try to use US hosting. Just from top of my head: 2itb.com, googlepages.com, freewebs.com...

    Quote Originally Posted by LovePimple
    Will the release version will have three (Fast, Normal, and Max) as you stated in your previous post?
    I think Quad will have just two modes. Two reasons:
    1. To simplify the source code.
    2. Normal mode is just a little bit slower than Fast mode.
    However, Ill think about Fast mode. For example, a new f option can be used, a la:
    quad ef 1.txt 1.quad

    (usage: quad {e[f|x]|d} <infile> <outfile>)

    or

    usage: quad <command> <infile> <outfile>
    commands:
    e = encode (default)
    f = fast mode
    x = max mode
    d = decode
    commands should be concatenated in any order, e.g. quad ex foo foo.quad

    What do you think?

  22. #22
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by encode
    usage: quad {e[f|x]|d} <infile> <outfile>
    If "e" means normal mode, "ef" fast and "ex" max mode, I vote for this option! Maybe just add some spaces like {e [f|x] | d}

    Quote Originally Posted by Stephan Busch
    And I am still searching for a free webhoster without traffic limit
    From what encode offered, googlepages.com seem to be the best choice, because it offers the most webspace (100MB) and its the only one from these three which doesnt seem to have bandwidth limit

  23. #23
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    Quote Originally Posted by Black_Fox
    If "e" means normal mode, "ef" fast and "ex" max mode, I vote for this option!
    Actually e means encode or compress. d - decode. And x - enable max mode.


  24. #24
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    I would prefer it like this:

    quad e <infile> <outfile> (Normal compression)

    quad ef <infile> <outfile> (If a Fast compression mode is added)

    quad ex <infile> <outfile> (Maximum compression)

  25. #25
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by Stephan Busch
    this seems to be a bit off-topic, but I am a bloody compiling beginner.. could someone please compile chile 0.4 and
    quicklz 1.20 for me?
    I would also like a compiled (Win32) copy of QuickLZ 1.20.

  26. #26
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by LovePimple
    quad e <infile> <outfile> (Normal compression)
    quad ef <infile> <outfile> (If a Fast compression mode is added)
    quad ex <infile> <outfile> (Maximum compression)
    This is exactly what I meant!

  27. #27
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    By the way, I sent a request to SourceForge.NET staff to reserve the QUAD project! Estimated release @ SF.NET - May-June 2007!

    QUAD will be released under GNU GPL license.

    Also, I think I'll keep improved version of QUAD v1.03a - with fast PPM instead of CM. In addition, QUAD at least will have two compression modes. It must be something like replacement for GZIP and BZIP2 file compressors. Of course, during this months a proper description/documentation must be written. I hope English-speakers will help me with that.

    A question (For users from US and UK ONLY):
    Should QUAD write 'compressing' instead of 'encoding' (and 'decompressing' instead of 'decoding')

    Some time ago, I start using this words, since they have equal length:
    encoding
    decoding

    Of course, commands must be changed to 'c' and 'd'

    With 'ratio' all okay, I guess. Or I can change it to something like:
    1159172 -> 492571
    or
    compressed 1159172 to 492571

    Another question about 'done'. AFAIK, DONE is more American word. Another possibilities: 'finished' - like with mcomp and all software from Malcolm Taylor.

    Any suggestions about 'last word'?

    Also what the difference between 'optimized' and 'optimised'? Is 'optimised' from New Zealand only?

    Also if you from UK or US, please post some suggestions about QUAD's output and documentation/description!

    Thank you!


  28. #28
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    No users from United States or United Kingdom? LovePimple I thought you're from England.

  29. #29
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    3,982
    Thanks
    377
    Thanked 351 Times in 139 Posts
    From the another side, such CLI is something like my signature. okay, now I'm thinking about which PPM include in Quad:

    1. An order-1 PPM with Full Exclusion
    2. A tricky order-2 PPM with Full Exclusion

    Speed is about the same in both cases. First, has some advantage on already compressed files. Second, has noticeable higher compression in most cases.


  30. #30
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by encode
    LovePimple I thought youre from England.
    English is not my native language, but I will do my best to help if you like. Im no expert though!

Page 1 of 2 12 LastLast

Similar Threads

  1. lzpm overview
    By encode in forum Forum Archive
    Replies: 4
    Last Post: 14th April 2007, 23:30
  2. QUAD 1.11 overview
    By encode in forum Forum Archive
    Replies: 45
    Last Post: 1st April 2007, 22:36
  3. QUAD 1.10 overview
    By encode in forum Forum Archive
    Replies: 10
    Last Post: 19th March 2007, 15:21
  4. QUAD v1.08 overview
    By encode in forum Forum Archive
    Replies: 40
    Last Post: 12th March 2007, 02:15
  5. QUAD 1.05a is here!
    By encode in forum Forum Archive
    Replies: 16
    Last Post: 5th March 2007, 00:22

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •