Page 3 of 6 FirstFirst 12345 ... LastLast
Results 61 to 90 of 179

Thread: RZM - a dull ROLZ compression engine

  1. #61
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,583
    Thanks
    234
    Thanked 160 Times in 90 Posts
    RZM Vs 7ZIP [LZMA]
    7ZIP 4.57 (Opt. -mx=9 - max 300 MB memory)
    24.864.490 Comp. 120,073 Seconds - Dec 4,400 Seconds.
    RZM v.0.06 (max 160 MB )
    24.684.208 Comp. 144,537 s. - Dec. 7,095 s.

  2. #62
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Another quick test...

    Test machine: Intel PIII (Coppermine) @750 MHz, 512 MB RAM, Windows 2000 Pro SP4

    Test File: ENWIK9 (1,000,000,000 bytes)

    Timed with AcuTimer v1.2


    ENWIK9 > 212,870,452 bytes
    Elapsed Time: 02:16:33.485 (8193.485 Seconds)

  3. #63
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 163 Times in 18 Posts
    Hi there!
    Thanks for the detailed tests (and for hosting the files, Lovepimple)! Here comes one more. Ive done some minor tweaks - this gives a slight improvement in overall compression. Your millage may vary depending on the input-data.

    Download 0.06b

    Quote Originally Posted by donkey7
    Christian
    i wonder if you will make ppm coder someday
    Maybe, but for now Ive no plans doing it.

  4. #64
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 163 Times in 18 Posts
    Quote Originally Posted by Nania Francesco Antonio
    To improve of few the speed of decompression I recommend you not to visualize the statistics during the operations (it also of 5-10%)
    Thanks for the clue. Ill give it a try, maybe it helps. But I visualize the status only every 64k or 128k - I dont remember.

  5. #65
    Member
    Join Date
    Jan 2008
    Posts
    33
    Thanks
    0
    Thanked 0 Times in 0 Posts
    quick test :

    machine: Intel Q6600 2.4Ghz , DDR800 2GB Ram , Windows Vista

    use AcuTimer v1.2 Copyright (c) 2007 by LovePimple


    RZM 0.06b (Mar 3 2008-64M-64K), copyright (c) C. Martelock

    Encoding 'enwik8' (using 258M memory)

    97656kb -> 23864kb (24437254b, 24.44%), done.


    Elapsed Time: 00 00:02:16.775 (136.775 Seconds)

    final size ---> 24,437,261 bytes

  6. #66
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by Christian
    Here comes one more. Ive done some minor tweaks - this gives a slight improvement in overall compression. Your millage may vary depending on the input-data.

    Download 0.06b
    Thanks Chris!

    Mirror: Download

  7. #67
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,583
    Thanks
    234
    Thanked 160 Times in 90 Posts
    SFC Test-> 11.453.693 bytes Comp. 43,327 s. - Dec. 3,812 s.

    C:>acutimer rzm c enwik8 t.a
    AcuTimer v1.2
    Copyright (c) 2007 by LovePimple
    RZM 0.06b (Mar 3 2008-64M-64K), copyright (c) C. Mar
    Encoding 'enwik8' (using 258M memory)
    97656kb -> 23864kb (24437254b, 24.44%), done.
    Elapsed Time: 00 00:02:25.190 (145,190 Seconds)


    C:>acutimer rzm d t.a test.txt
    AcuTimer v1.2
    Copyright (c) 2007 by LovePimple
    RZM 0.06b (Mar 3 2008-64M-64K), copyright (c) C. Mar
    Decoding 't.a' (using 130M memory)
    23864kb -> 97656kb (100000000b, 409.21%), done.
    Elapsed Time: 00 00:00:06.890 (6,890 Seconds)

  8. #68
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,583
    Thanks
    234
    Thanked 160 Times in 90 Posts
    I have tried, for game, to compress MOC test with RZM adding a program (based on Rings 1.1 filters ) pre-filter for the images BMP,TIFF..WAVE. etc
    My result has been -> 147.534.234 bytes (time 269,763 sec.) ! Very Very Very nice!

  9. #69
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by Nania Francesco Antonio
    I have tried, for game, to compress MOC test with RZM adding a program (based on Rings 1.1 filters ) pre-filter for the images BMP,TIFF..WAVE. etc
    My result has been -> 147.534.234 bytes (time 269,763 sec.) ! Very Very Very nice!

  10. #70
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 163 Times in 18 Posts
    Quote Originally Posted by Nania Francesco Antonio
    I have tried, for game, to compress MOC test with RZM adding a program (based on Rings 1.1 filters ) pre-filter for the images BMP,TIFF..WAVE. etc
    My result has been -> 147.534.234 bytes (time 269,763 sec.) ! Very Very Very nice!
    Thanks for the interesting test. Do you use delta coding (and channel seperation), or are the filters more advanced?

    New day, new version. Only little adjustments.
    RZM 0.06c

  11. #71
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,583
    Thanks
    234
    Thanked 160 Times in 90 Posts
    Delta coding (and channel seperation)! Hi!

  12. #72
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    926
    Thanks
    58
    Thanked 116 Times in 93 Posts
    Unreal Tournament 2004 CD6 in MDF/MDS format
    then ran it through ECM, Precomp and rep


    filename/method - size -decomp time
    Originale - 615.577.164 bytes - 0s
    7-zip ultra 256mb 273 - 584.899.521 bytes - 54s
    RZM 0.6 - 580.128.828 bytes - 121s

    So RZM beats 7-zip in compression, but decompression is more than twice the time of 7-zip

  13. #73
    Tester
    Nania Francesco's Avatar
    Join Date
    May 2008
    Location
    Italy
    Posts
    1,583
    Thanks
    234
    Thanked 160 Times in 90 Posts
    RZM 0,06C
    SFC test
    11.431.947 b. COMP. 42,797 s. - DEC. 3,844 s.
    enwik8 test
    97656kb -> 23857kb (24429590b, 24.43%), done.
    Elapsed Time: 00 00:02:24.224 (144,224 Seconds)
    UCLC Pgm files test
    5.657.858 b. COMP. 5,938 s. - DEC. 2,516 s.
    CALGARY Corpus test
    821.321 b. COMP. 2,704 s. - DEC. 0,389 s.

  14. #74
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 163 Times in 18 Posts
    Quote Originally Posted by SvenBent
    7-zip ultra 256mb 273 - 584.899.521 bytes - 54s
    RZM 0.6 - 580.128.828 bytes - 121s

    So RZM beats 7-zip in compression, but decompression is more than twice the time of 7-zip
    Thanks for the detailed test, Sven. Yepp, decompression of RZM is slower - though, the difference gets less, the higher the compression ratio. Still, imo decompression speed is very practical. But even if compression is slightly higher, RZM is experimental, only a few days old and has no frozen syntax, ... So, if youre looking for an archiving tool to backup your CD collection, maybe you should go with 7-zip.

    Quote Originally Posted by Nania Francesco Antonio
    Delta coding (and channel seperation)! Hi!
    So, maybe we can achieve even better results with advanced transforms.

  15. #75
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by Christian
    New day, new version. Only little adjustments.
    RZM 0.06c
    Thanks Chris!

    Mirror: Download

  16. #76
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Thank you! RZM has been tested preliminarily on my usual testset:
    12 367 319 - RZM 0.06c - EDIT: missed 0.06c originally
    12 370 286 - RZM 0.06b
    12 502 511 - RZM 0.02
    11 982 518 - CCM 1.30a

  17. #77
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Christian, great to have you back! I really like RZM. Can't wait until you add those extra filters and really make it shine. Do you pull some of the same "cache tricks" that you used in ccm in rzm? Also, do you plan on doing any more work on CCM, or are you getting ready to make some new compressor that combines ideas from both RZM and CCM?

    Anyway, here are some tests:

    Test System: AMD Athlon 64 3000+ (Venice, single core) with 2GIG Corsair DDR CL2 RAM

    rzm006b->rzm.exe c enwik8 (24437261 Bytes) - 3 m 14 s -
    rzm006b->rzm.exe d enwik8 - 0 m 11 s -
    rzm006c->rzm.exe c enwik8 (24429597 Bytes) - 3 m 14 s - slightly smaller
    rzm006c->rzm.exe d enwik8 - 0 m 11 s - No Change

    rzm006b->rzm.exe c enwik9 (210722103 Bytes) - 32 m 20 s -
    rzm006b->rzm.exe d enwik9 - 1 m 52 s -
    rzm006c->rzm.exe c enwik9 (210719085 Bytes) - 32 m 19 s - slightly faster slightly smaller
    rzm006c->rzm.exe d enwik9 - 1 m 35 s - decompression 17.8% faster!

  18. #78
    Member
    Join Date
    Sep 2007
    Location
    Denmark
    Posts
    926
    Thanks
    58
    Thanked 116 Times in 93 Posts
    Quote Originally Posted by Christian
    So, if youre looking for an archiving tool to backup your CD collection, maybe you should go with 7-zip.
    I always do a md5 comparison of the decompressed file.
    And I keep the decompresser(s) within the final 7-zip container.
    just like with CCM(x) and PackJPG, to avoid incompatibilities due to changed format/specs.


    and yes the decompression time it very practical. especially when you consider the possibility of Multithreading decompressing by using batch rutines for several files.
    that way the 6 CDs only take 50% (roghtly) more time on a quadcore cpu.

  19. #79
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 163 Times in 18 Posts
    Quote Originally Posted by Hahobas
    Cant wait until you add those extra filters and really make it shine.
    Thanks to the great feedback, I seriously consider adding filters. But still, theyll only improve (sometimes hurt) compression on pure RGB, WAVE or similar data structures.
    Maybe Ill add table detection, too - but Im not so sure if its worth it. Bulat, if you read, do you have any numbers regarding the compression gain expected from table detection?

    Quote Originally Posted by Hahobas
    Do you pull some of the same "cache tricks" that you used in ccm in rzm?
    Of course, Im using some tricks Ive learned during the development of CCM. But frankly, since RZM is LZ-based, it doesnt have much in common with CCMs architecture.

    Quote Originally Posted by Hahobas
    Also, do you plan on doing any more work on CCM
    I still want to add PreComp to CCM. Besides that, maybe. The list with ideas and improvements to CCM got rather extensive - at some point I lost focus/interest.

    Quote Originally Posted by Hahobas
    getting ready to make some new compressor that combines ideas from both RZM and CCM
    I thought about that. But Ive not made up my mind yet. If you combine a very strong symbol-coder with LZ, you end up using the symbol-coder most of the time - which makes the LZ-part useless. Therefore, imo the symbolcoder should only try to model things, the LZ-part cant.

  20. #80
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 163 Times in 18 Posts
    Quote Originally Posted by SvenBent
    I always do a md5 comparison of the decompressed file.
    And I keep the decompresser(s) within the final 7-zip container.
    just like with CCM(x) and PackJPG, to avoid incompatibilities due to changed format/specs.
    You are doing your backups very thoroughly. In that case, you have one more choice with RZM.

    Quote Originally Posted by SvenBent
    that way the 6 CDs only take 50% (roghtly) more time on a quadcore cpu.
    Very true. This must be one of the scenarios, in which a quadcore is of some real value (as long as IO isnt the bottleneck).

  21. #81
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 163 Times in 18 Posts
    Quote Originally Posted by Black_Fox
    Thank you! RZM has been tested preliminarily on my usual testset:
    12 367 319 - RZM 0.06c - EDIT: missed 0.06c originally
    12 370 286 - RZM 0.06b
    12 502 511 - RZM 0.02
    11 982 518 - CCM 1.30a
    Thanks a lot for the benchmark, BlackFox! Looking at those numbers, the syntax improvement really payed off.

  22. #82
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    If the BMP filter will have similar effectivity as that one in CCM, RZM will have better ratio than CCM!

  23. #83
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 163 Times in 18 Posts
    I don't remember CCM's size on your benchmark without any filtering (iirc, 1.1.2a was the last one without filtering). If CCM started around 850.000 and achieved ~745.000 with filtering applied, than RZM will most probable gain only 80-120k, too.

  24. #84
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,593
    Thanks
    801
    Thanked 698 Times in 378 Posts
    Quote Originally Posted by Christian
    Bulat, if you read, do you have any numbers regarding the compression gain expected from table detection?
    2% for very average binary file, 1mb for MFC, take into account that these numbers are for very sophisticated preprocessor, found only in rar&fa

    you can test it youself, applying only table preprocessor to the data:
    arc a a file -m=table --nodir

  25. #85
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 163 Times in 18 Posts
    Thanks Bulat! Ill try it. So, I think Ill add table detection then, too. But before next weekend theres no time I can spare on RZM.

    From the FreeArc thread:
    Quote Originally Posted by Bulat Ziganshin
    [External compressor:rzm]
    cmem = 256
    dmem = 64
    packcmd = rzm c $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
    unpackcmd = rzm d $$arcpackedfile$$.tmp $$arcdatafile$$.tmp

    im not sure about mem settings..
    RZM uses ~258MB/~130MB for compression/decompression. The sliding Window is 64MB, but during decompression ROLZ needs additional 64M (256*64K*4) for the offsets, too.

  26. #86
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Christian
    I dont remember CCMs size on your benchmark without any filtering (iirc, 1.1.2a was the last one without filtering). If CCM started around 850.000 and achieved ~745.000 with filtering applied, than RZM will most probable gain only 80-120k, too.
    Youre right, I have been mistaken a bit - CCM 1.1.1a scores ~950k, then there are two more versions inbetween featuring only small hashing and modelling tweaks, then comes 1.1.2a with ~815k, which still doesnt use any filters It seems I have never tested 1.15a, but it scores about 1000 bytes smaller than 1.1.2a... So it may be a bit less for mm filters. Total ratio is nevertheless brilliant for an assymetric coder, when compared to CCM.

  27. #87
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,593
    Thanks
    801
    Thanked 698 Times in 378 Posts
    Quote Originally Posted by Bulat Ziganshin
    arc a a file -m=table --nodir
    arc a a file -m=delta --nodir

    Quote Originally Posted by Christian
    So, I think Ill add table detection then, too.
    unless you are going to use my code, your improvement will be smaller. ive spent several weeks on this engine

  28. #88
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,023
    Thanks
    415
    Thanked 416 Times in 158 Posts
    Quote Originally Posted by Christian
    Parsing is not much different from LZ77 parsing. I do forward parsing - it makes state propagation and string searching easier. I dont know what you want to know exactly?
    Quote Originally Posted by Christian
    As I dont want to reinvent the wheel, Ive just searched and found a nice post by Bulat. It describes the ideas of OP very well. For ROLZ it isnt any different - only the matches come from a different source.
    Click me
    One thing, you say that you use forward parsing - and in Bulat description we use backwards parsing.
    In short, SS parsing can be defined as follows:
    1. Find longest matches at each block position
    2. Parse backwards, keeping sum of bits needed to encode each blocks position from end. We check each position, and if we find that better to encode literal sequence here, we drop the match.

    Can you describe yours?

  29. #89
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by encode
    Can you describe yours?
    Ill be very happy, too
    BIT Archiver homepage: www.osmanturan.com

  30. #90
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 163 Times in 18 Posts
    Quote Originally Posted by encode
    One thing, you say that you use forward parsing - and in Bulat description we use backwards parsing.
    In short, SS parsing can be defined as follows:
    ...
    Can you describe yours?
    Just read the description of bit-optimal parsing again. The cost optimization is done forward in this description. Only the final path construction is done backwards.

Page 3 of 6 FirstFirst 12345 ... LastLast

Similar Threads

  1. BALZ - An Open-Source ROLZ-based compressor
    By encode in forum Data Compression
    Replies: 60
    Last Post: 6th March 2015, 17:47
  2. RZM doesn't like to share cpu
    By SvenBent in forum Data Compression
    Replies: 2
    Last Post: 19th July 2008, 23:35
  3. ROLZ explanation?
    By Trixter in forum Data Compression
    Replies: 5
    Last Post: 10th June 2008, 19:24
  4. RZM compressor
    By encode in forum Data Compression
    Replies: 2
    Last Post: 24th May 2008, 15:59
  5. A small article on ROLZ (Russian)
    By encode in forum Forum Archive
    Replies: 21
    Last Post: 29th April 2007, 16:18

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •