Page 6 of 6 FirstFirst ... 456
Results 151 to 179 of 179

Thread: RZM - a dull ROLZ compression engine

  1. #151
    Member
    Join Date
    Feb 2009
    Location
    USA
    Posts
    58
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Christian
    Actually, for a context mixer whose inner loop has a couple of branches, its quite easy to predict the branching behaviour.
    Yeah, with CCM I bet that is the case. Looking forward to a new version, and if youre in the mood we could try a 64bit version . If you dont have a 64bit machine you could just use QEMU and run a 64bit VM. Id be happy to test the 32bit vs 64.

    Im glad youre not giving up on CCM, its by far my favorite compressor.

  2. #152
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Do you have any other tips for us? I was quiet surprised how less i know about gcc's switches and things like this - what a shame, i'm using gcc for years...

    BTW: Christian, how do you model bits? Do you use any mapping like paq's states, or do you directly model bits by counters?
    M1, CMM and other resources - http://sites.google.com/site/toffer86/ or toffer.tk

  3. #153
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 155 Times in 18 Posts
    Quote Originally Posted by toffer
    Do you have any other tips for us? I was quiet surprised how less i know about gccs switches and things like this - what a shame, im using gcc for years...
    Same here - I tried a few switches and always lost interest after a couple of minutes.


    toffer
    I use small state machines - should be similar to paq states.

  4. #154
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    One of the things on my todo list is to generate several state machines for different orders. Do you have any experience with this? I did a state machine, which is more efficient than paq's at order 0. But i don't want to waste my time in doing this for every order...
    M1, CMM and other resources - http://sites.google.com/site/toffer86/ or toffer.tk

  5. #155
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 155 Times in 18 Posts
    Quote Originally Posted by toffer
    One of the things on my todo list is to generate several state machines for different orders. Do you have any experience with this?
    I tried, but I rejected the approach because improvements were a mixed bag - and I wanted to keep the code small. Additionally, mixing/SSE cushions the effect, too.

    Good night.

  6. #156
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,004
    Thanks
    393
    Thanked 389 Times in 149 Posts
    By the way, Criss, do you plan to release the source of your compressors? (CCM, RZM, SLUG,...)

  7. #157
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 155 Times in 18 Posts
    No plans anymore. For me, it's a must to have the source in a nice state before releasing it. But e.g., I never even 'cleaned' CCM's source because I already moved on to RZM and stuff. There are so many OS programs whose source code is a living hell - I don't want to contribute to that. Additionally, I do not think, CCM's source would be of great use. Most people would just rip some ideas out of it and that's it.
    Currently I'm working on something new, which is not 'directly' related to compression. So, again no spare time to tidy up...

  8. #158
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Quote Originally Posted by Christian
    No plans anymore. For me, its a must to have the source in a nice state before releasing it. But e.g., I never even cleaned CCMs source because I already moved on to RZM and stuff. There are so many OS programs whose source code is a living hell - I dont want to contribute to that. Additionally, I do not think, CCMs source would be of great use. Most people would just rip some ideas out of it and thats it.
    Currently Im working on something new, which is not directly related to compression. So, again no spare time to tidy up...
    As i said in some other post i mostly like the idea of information interchange (not the copy and paste thing...)
    M1, CMM and other resources - http://sites.google.com/site/toffer86/ or toffer.tk

  9. #159
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,537
    Thanks
    758
    Thanked 676 Times in 366 Posts
    i'm interested in using rzm in freearc. how about releasing it under GPL license?

  10. #160
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,004
    Thanks
    393
    Thanked 389 Times in 149 Posts
    Quote Originally Posted by toffer
    As i said in some other post i mostly like the idea of information interchange (not the copy and paste thing...)
    ...or paper/post describing some ideas and principles...

  11. #161
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 155 Times in 18 Posts
    Quote Originally Posted by toffer
    As i said in some other post i mostly like the idea of information interchange (not the copy and paste thing...)
    I like information interchange much better than copy&paste, too. Insight is much better than blind trust.


    Quote Originally Posted by encode
    ...or paper/post describing some ideas and principles...
    Honestly, I work on data compression one or two hours a week - as a hobby, of course. Sometimes more and most of the time much less. But I can think of many things more interesting/important than writing a paper about my compressors. Btw., I recall, that youve planned writing a paper about ROLZ yourself. But still, you can send me mail, if you have some questions.


    Quote Originally Posted by Bulat Ziganshin
    how about releasing it under GPL license?
    I have no plans for that.

    Btw., for now, work on CCM/RZM/... is frozen. I just cant stand all those requests I get. Maybe I have to get a thicker skin for data compression.

  12. #162
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,004
    Thanks
    393
    Thanked 389 Times in 149 Posts
    Quote Originally Posted by Christian
    Btw., I recall, that youve planned writing a paper about ROLZ yourself.
    Its done some time ago (in Russian though):
    http://ru.wikipedia.org/wiki/ROLZ


  13. #163
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 155 Times in 18 Posts
    Here's a new compilation of RZM. Nothing really new, but I moved to GCC 4.3.0 using profile optimization. Thanks again to Hahobas (and of course the GCC guys) who pointed my at this technique. It might be a tiny tiny bit faster - or not. Just try it out.

    RZM 0.07f

  14. #164
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 155 Times in 18 Posts
    One more new compile. Acutally, this includes a little bugfix, too. Now, memory allocation for compression really is ~258M. Additionally, compression and decompresssion speed are slightly faster than 0.07e.

    Download RZM 0.07h

  15. #165
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,537
    Thanks
    758
    Thanked 676 Times in 366 Posts
    0.07e?

  16. #166
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,537
    Thanks
    758
    Thanked 676 Times in 366 Posts
    this new site prohibits downloaders from Russia

  17. #167
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 155 Times in 18 Posts
    So, I'll upload all files to Rapidshare and their funny cats ... again.
    But this has to wait for tomorrow - perhaps LovePimple can host the files before that.

  18. #168
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 155 Times in 18 Posts
    Quote Originally Posted by Bulat Ziganshin
    this new site prohibits downloaders from Russia
    Maybe you can try to use a proxy. Anyway, sorry for the inconvenience.

  19. #169
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,004
    Thanks
    393
    Thanked 389 Times in 149 Posts
    Try out the Send Space:
    http://www.sendspace.com/


  20. #170
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,537
    Thanks
    758
    Thanked 676 Times in 366 Posts
    i don't had more luck with rapidshare because there are many thousands users sharing the same IP due to our provider's NAT

    actually, i'm not hungry about others' programs and LovePimple page is enough for me, i noted it just to inform you that it may be inconvenient for other users

  21. #171
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,537
    Thanks
    758
    Thanked 676 Times in 366 Posts
    Quote Originally Posted by encode
    http://www.sendspace.com/
    successfully tried to upload & dl file here

  22. #172
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks Chris!

    Mirror: Download

  23. #173
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    How about a googlepages web? Shelwien is using that and there was no complaint so far. Additionaly, no dogs, cats, timeouts and whatsoever

  24. #174
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 155 Times in 18 Posts
    RZM now has a homepage to circumvent the web hosting nuisance.

    Thanks for clue the BlackFox. The latest problems finally made me create a small home page. Curiously, googlepages is very slow for me - although, the homepage is like 2 or 3 kb.

  25. #175
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,004
    Thanks
    393
    Thanked 389 Times in 149 Posts
    By the way, Criss, can you explain how do you perform a string search? Do you use hashing or BT? And how do you deal with ROLZ nature? As you posted above, you keep an offset table:
    offsets[win[i-1]][...]
    Just thinking about an improved ROLZ scheme. With LZPM, I use hash chains, to determine the match index I keep the char count in the hash element with actual offset. If we will add a phrase per each byte, I think we may simplify the search...

  26. #176
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 155 Times in 18 Posts
    Quote Originally Posted by encode
    By the way, Criss, can you explain how do you perform a string search? Do you use hashing or BT?
    In RZM I use binary trees. Of course, Slug uses hashing for speed reasons.
    I described the idea of string-insertion for binary trees in the tornado 0.4 topic. If you understand it then string search is easy. Since RZM uses optimal parsing it needs to search at each position - therefore, you just search while inserting the string for the current position.

    Quote Originally Posted by encode
    And how do you deal with ROLZ nature? As you posted above, you keep an offset table:
    offsets[win[i-1]][...]
    The size of each table is fixed (as it is hinted on Slugs homepage) to 64K entries. RZM adds one offset for each step - additionally it keeps some long distance offsets for better handling of compressed data.

    Quote Originally Posted by encode
    If we will add a phrase per each byte, I think we may simplify the search...
    Is this related to hash chaining? If not, what exactly do you mean?

  27. #177
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,004
    Thanks
    393
    Thanked 389 Times in 149 Posts
    Quote Originally Posted by Christian
    I described the idea of string-insertion for binary trees in the tornado 0.4 topic.
    OK, will carefully read this topic.

    Quote Originally Posted by Christian
    Is this related to hash chaining?
    Yes, Im just thinking, how we can translate the real offsets into match indexes. In other words, we may use pure LZ77 with MINMATCH=4 (as BALZ v1.04) and then drop first byte of a match as a literal and encode match length of three, plus, an offset as index. Generally speaking, LZPM does it exactly in such way, but for offset->index translation LZPM keeps byte counts - i.e. cnt[c] - keeps count of cs. Thus, match index = cnt[c]-node.cnt;

    Although, you may test the latest BALZ, its not that worser than LZPM, in most cases its even better, especially on binary data. So, now Im thinking about further improving LZ77 scheme especially in offset coding, maybe better buffered offsets technique, etc.

  28. #178
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,537
    Thanks
    758
    Thanked 676 Times in 366 Posts
    Christian, is rzm still limited to files <2gb?

    and - how about adding faster lazy-parsing compression mode?

  29. #179
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 155 Times in 18 Posts
    Quote Originally Posted by Bulat Ziganshin
    Christian, is rzm still limited to files <2gb?
    Yes.

    Quote Originally Posted by Bulat Ziganshin
    and - how about adding faster lazy-parsing compression mode?
    Great idea, but sadly - no time. And originally, RZM is about strong asymmetric compression.

Page 6 of 6 FirstFirst ... 456

Similar Threads

  1. BALZ - An Open-Source ROLZ-based compressor
    By encode in forum Data Compression
    Replies: 60
    Last Post: 6th March 2015, 16:47
  2. RZM doesn't like to share cpu
    By SvenBent in forum Data Compression
    Replies: 2
    Last Post: 19th July 2008, 22:35
  3. ROLZ explanation?
    By Trixter in forum Data Compression
    Replies: 5
    Last Post: 10th June 2008, 18:24
  4. RZM compressor
    By encode in forum Data Compression
    Replies: 2
    Last Post: 24th May 2008, 14:59
  5. A small article on ROLZ (Russian)
    By encode in forum Forum Archive
    Replies: 21
    Last Post: 29th April 2007, 15:18

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •