Results 1 to 30 of 35

Thread: fcm1 - open source order-1 cm encoder

Hybrid View

Previous Post Previous Post   Next Post Next Post
  1. #1
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,000
    Thanks
    387
    Thanked 365 Times in 145 Posts

    Talking fcm1 - open source order-1 cm encoder

    OK, let me introduce some encoder. Continuing fpaq0f2-like, experimental encoders, I made a new one - check out my variant of an order-1 cm. Source code included:
    fcm1.zip


  2. #2
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,000
    Thanks
    387
    Thanked 365 Times in 145 Posts

    Cool

    Brief description:

    Each counter consists from a two states a la FPAQ0P - fast and slow one, which are summarized together.

    For mixing a probabilities from order-1 and order-0 we use followed formula:

    ((p2*3)+p1)/4

    p2 - Probability from an order-1 context
    p1 - Probability from an order-0 context

    If:
    p2 = 0.5
    i.e. order-1 context for the first time occurs, we use probability from an order-0 only.

    Any thoughts?

    EDIT:
    In final version, we use:
    ((p2*15)+p1)/16

  3. #3
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts

    Thumbs up

    Quote Originally Posted by encode View Post
    OK, let me introduce some encoder. Continuing fpaq0f2-like, experimental encoders, I made a new one - check out my variant of an order-1 cm. Source code included:
    fcm1.zip

    Thanks Ilia! I have uploaded it to my site along with my own speed optimised compile.

    Mirror: Download

  4. #4
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    One can't generalize this, but you give far too much weight to order 0. The 2d SSE visualizations (SSE for mixing) in the old form showed that the SSE function mostly looked like a plane, which was dominated by the order 1 model. That means, order 0 gave almost no contribution.
    How much does your slow counter/fast counter mix improve over a standalone counter?

  5. #5
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,000
    Thanks
    387
    Thanked 365 Times in 145 Posts
    Quote Originally Posted by toffer View Post
    How much does your slow counter/fast counter mix improve over a standalone counter?
    Notable difference... You may check it for yourself!

  6. #6
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,000
    Thanks
    387
    Thanked 365 Times in 145 Posts
    Updated archive with a new compile which is at almost 2X times faster!

  7. #7
    Programmer toffer's Avatar
    Join Date
    May 2008
    Location
    Erfurt, Germany
    Posts
    587
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Hm as i said giving order 0 such a high weight is totally wrong:

    -> 22.459.785 bytes (plain order1) vs. 23.783.905 bytes (orders01) on SFC

    I just changed this:

    Code:
    //  int P() {
    //    int p1=counter0[c0].P();
    //    int p2=counter1[c1][c0].P();
    //
    //    return (p2!=(1<<16)?((p2*3)+p1)>>7:p1>>5);
    //  }
    
      int P() {
        return counter1[c1][c0].P()>>5;
      }

  8. #8
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,000
    Thanks
    387
    Thanked 365 Times in 145 Posts
    I'm not so sure. It's a matter of tuning for specific file type and specific situation...

    Static weights form PAQ1 are: 4, 9, 16, ...

    sfc.7z:
    p2*3: 23,789,646 bytes
    p2*7: 22,869,851 bytes
    p2*15: 22,535,420 bytes
    plain order-1: 22,487,664 bytes
    p2*31: 22,425,582 bytes
    p2*63: 22,403,145 bytes
    ...


  9. #9
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Quote Originally Posted by encode View Post
    Updated archive with a new compile which is at almost 2X times faster!
    Quote Originally Posted by toffer View Post
    Hm as i said giving order 0 such a high weight is totally wrong:

    -> 22.459.785 bytes (plain order1) vs. 23.783.905 bytes (orders01) on SFC

    I just changed this:

    Code:
    //  int P() {
    //    int p1=counter0[c0].P();
    //    int p2=counter1[c1][c0].P();
    //
    //    return (p2!=(1<<16)?((p2*3)+p1)>>7:p1>>5);
    //  }
    
      int P() {
        return counter1[c1][c0].P()>>5;
      }
    Thanks Ilia, and toffer! I have updated my archive.

    Mirror: Download

Similar Threads

  1. Replies: 23
    Last Post: 24th March 2018, 18:57
  2. BALZ - An Open-Source ROLZ-based compressor
    By encode in forum Data Compression
    Replies: 60
    Last Post: 6th March 2015, 17:47
  3. PeaZip - open source archiver
    By squxe in forum Data Compression
    Replies: 1
    Last Post: 3rd December 2009, 22:01
  4. New fast open-source paq-based jpeg compressor
    By Bulat Ziganshin in forum Forum Archive
    Replies: 14
    Last Post: 13th September 2007, 14:57
  5. Replies: 0
    Last Post: 26th July 2007, 19:47

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •