Page 2 of 3 FirstFirst 123 LastLast
Results 31 to 60 of 86

Thread: mod_ppmd

  1. #31
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,237
    Thanks
    192
    Thanked 968 Times in 501 Posts
    So here I replaced the rangecoder in paq8p, as a demo:
    http://nishi.dreamhosters.com/u/paq8p_rc_v0.rar (sources included)

    The actual problem is caused by this line in paq rc:
    while (((x1^x2)&0xff000000)==0) {  // pass equal leading bytes of range
    It flushes an extra byte to avoid x1 overflow, but this basically means
    an extra output byte for each long enough series of zero bits, especially
    if the model assigns low probability to zero.

    There're also some other minor precision issues, but that was the main one.
    It could be important for image compression.

    Code:
          orig.    paq8p  "paq8q"
    zero  1048576     397     74 // 1M of \x00
    ones  1048576     402     31 // 1M of \xFF
    book1  768771  192296 192275 // book1 from calgary corpus
    w.bmp 1440054  243409 243262 // some "winpe.bmp" file

  2. The Following 2 Users Say Thank You to Shelwien For This Useful Post:

    mpais (5th February 2017),xinix (1st April 2017)

  3. #32
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    235
    Thanks
    102
    Thanked 140 Times in 102 Posts
    During my tests with mod_ppmd_v4_dll.rar I found that ppmd_Init(..., 16, 64, 1) with ppmd_Setbit(..., !bit) gives the Windows error "... has stopped working" at position 41983526 of ENWIK8.
    You can replicate the error with the following program:
    // g++ (x86_64-win32-sjlj-rev0, Built by MinGW-W64 project) 5.2.0
    // g++ -opmd.exe pmd.cpp modppmd64.dll && pmd < ENWIK8

    #include <stdio.h>
    #include <stdlib.h>
    #include "modppmd.h"

    #define SCALE 32767

    int main( int argc, char** argv ) {
    void *ppmd = ppmd_Alloc();
    ppmd_Init(ppmd, 16, 64, 1);

    for(int i = 0, c = 0; (c = getc(stdin)) != EOF; i++ ) {
    if (!(i & 0xffff) || (i >= 41983526 - 4)) printf("%12i %8.3f\r", i, i / 1048576.0);

    ppmd_Predict(ppmd, SCALE); ppmd_Setbit(ppmd, !(c & 0x80));
    ppmd_Predict(ppmd, SCALE); ppmd_Setbit(ppmd, !(c & 0x40));
    ppmd_Predict(ppmd, SCALE); ppmd_Setbit(ppmd, !(c & 0x20));
    ppmd_Predict(ppmd, SCALE); ppmd_Setbit(ppmd, !(c & 0x10));
    ppmd_Predict(ppmd, SCALE); ppmd_Setbit(ppmd, !(c & 0x08));
    ppmd_Predict(ppmd, SCALE); ppmd_Setbit(ppmd, !(c & 0x04));
    ppmd_Predict(ppmd, SCALE); ppmd_Setbit(ppmd, !(c & 0x02));
    ppmd_Predict(ppmd, SCALE); ppmd_Setbit(ppmd, !(c & 0x01));
    }

    ppmd_Quit(ppmd);
    ppmd_Free(ppmd);

    return 0;
    }

    Some questions:
    - Is it possible to have a function that returns the max order found by PPMd?
    - In ppmd_Setbit(), must the bit value be 1 or != 0 is enough?
    - Is there any reason that we must pass SCALE every call to ppmd_Predict() instead of pass it, e.g., only in ppmd_Init()?

  4. The Following User Says Thank You to Mauro Vezzosi For This Useful Post:

    Shelwien (8th March 2017)

  5. #33
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,237
    Thanks
    192
    Thanked 968 Times in 501 Posts
    > error "... has stopped working" at position 41983526 of ENWIK8.

    Thanks a lot, it actually affects ppmd_sh and all derivatives.

    http://nishi.dreamhosters.com/u/mod_ppmd_v4_dll.rar

    I also added the source, so you don't need to use the dll, just link with modppmd.cpp .
    Or include libpmd.inc directly, then you'd be able to access OrderFall field in Model class.

    > #define SCALE 32767

    Is this intentional? Normally its power-of-2 (4096 in paq).

    > - Is it possible to have a function that returns the max order found by PPMd?

    Yes, I'd add it a bit later.

    I'd also make SSE optional.

    > - In ppmd_Setbit(), must the bit value be 1 or != 0 is enough?

    Yes, its added to contexts as ctx+=ctx+y;

    > Is there any reason that we must pass SCALE every call to ppmd_Predict()
    > instead of pass it, e.g., only in ppmd_Init()?

    It seemed natural, because that scale value is only used in ppmd_Predict.
    I can move it to Init if necessary, but don't really see the point.
    And its even possible to invent some obscure cases where we might want
    to change it in runtime.

  6. The Following 3 Users Say Thank You to Shelwien For This Useful Post:

    Mauro Vezzosi (10th March 2017),RamiroCruzo (10th March 2017),xinix (10th March 2017)

  7. #34
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    225
    Thanks
    106
    Thanked 106 Times in 65 Posts
    Quote Originally Posted by Shelwien View Post
    > error "... has stopped working" at position 41983526 of ENWIK8.

    Thanks a lot, it actually affects ppmd_sh and all derivatives.

    http://nishi.dreamhosters.com/u/mod_ppmd_v4_dll.rar
    Hi Shelwien, does this bug affect the version of ppmd used in cmix? If so, do you have any instructions on how I can merge in the fix? Thanks!

  8. #35
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,237
    Thanks
    192
    Thanked 968 Times in 501 Posts
    @byronknoll:
    Yes, at https://github.com/byronknoll/cmix/b.../ppmd.cpp#L761 insert "pc = MaxContext;"

    Btw, I was able to improve cmix result for enwik8 to 15427xxx using SSE like in mod_ppmd_v4.
    And even a bit better using external mix with another mod_ppmd.
    (Actually maybe 15422xxx, because I was only able to compress enwik8 with cmix to 15445xxx, not 440 like listed).

  9. The Following 2 Users Say Thank You to Shelwien For This Useful Post:

    byronknoll (11th March 2017),xinix (1st April 2017)

  10. #36
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    235
    Thanks
    102
    Thanked 140 Times in 102 Posts
    > I also added the source, so you don't need to use the dll, just link with modppmd.cpp .
    > Or include libpmd.inc directly, then you'd be able to access OrderFall field in Model class.
    >> - Is it possible to have a function that returns the max order found by PPMd?
    > Yes, I'd add it a bit later.
    > I'd also make SSE optional.
    Thanks for all.

    >> #define SCALE 32767
    > Is this intentional? Normally its power-of-2 (4096 in paq).
    You are right, I was wrong to write 32767 in the program of the post, my intention was to write 32768.

  11. #37
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    225
    Thanks
    106
    Thanked 106 Times in 65 Posts
    Quote Originally Posted by Shelwien View Post
    @byronknoll:
    Yes, at https://github.com/byronknoll/cmix/b.../ppmd.cpp#L761 insert "pc = MaxContext;"

    Btw, I was able to improve cmix result for enwik8 to 15427xxx using SSE like in mod_ppmd_v4.
    And even a bit better using external mix with another mod_ppmd.
    (Actually maybe 15422xxx, because I was only able to compress enwik8 with cmix to 15445xxx, not 440 like listed).
    Thanks! I'll merge in the fix soon.

    Wow, that is a nice improvement with SSE! The last time I checked, the SSE I implemented in cmix only added a *tiny* improvement, so maybe I have some bug in my implementation. I will take a look at the SSE code in mod_ppmd_v4!

    Let me know what results you get with adding additional mod_ppmd models. The cmix byte-level mixer (using LSTM) is extremely slow, so adding models there has a huge performance penalty. However, it can be added to the bit-level mixer with low cost. I did experiment with adding multiple mod_ppmd models in an earlier cmix version, but didn't find a good performance vs compression gain tradeoff.

  12. #38
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,237
    Thanks
    192
    Thanked 968 Times in 501 Posts
    > The last time I checked, the SSE I implemented in cmix only added a *tiny* improvement,
    > so maybe I have some bug in my implementation.

    1. Its my own interpolated SSE implementation, with backpropagation update.
    Its quite different from paq ones.
    2. I'm optimizing the parameters.

    > I will take a look at the SSE code in mod_ppmd_v4!

    Here I uploaded all the related sources: http://nishi.dreamhosters.com/u/cmix_SSE_v0.rar
    Code:
    768771 book1         // book1 = data bits extracted from book1cmix1.pd
    186094 book1orig.ari // book1cmix1.pd without SSE               
    185860 book1s.ari    // book1cmix1.pd with external SSE         
    185581 book1sm.ari   // book1cmix_pmd.pd with external SSE/mixer
    These are results for unpreprocessed book1, also optimization profiles are different from enwik8 ones
    (kampaster still runs an optimizer for these and I don't have final profiles).
    But models are the same, and there should be some improvement on enwik8 cmix .pd too with these coders.
    (Maybe 15430xxx or so).

    > The cmix byte-level mixer (using LSTM) is extremely slow,
    > so adding models there has a huge performance penalty.

    I'd suggest rearranging it like I did for this SSE experiment.
    You can actually write probability dumps from various submodels -
    then you'd only need to recompute parts that actually change.

    > I did experiment with adding multiple mod_ppmd models in an earlier cmix version,
    > but didn't find a good performance vs compression gain tradeoff.

    I suspect that you get skewed weight values for ppmd.
    Consider mix(ppmd,paq) vs mix(ppmd,paq,paq) - prediction becomes worse in 2nd case,
    because paq estimation effectively gets 2x weight multiplier, so ppmd estimation has less effect.

    Unlike cmix and emma, I barely got any improvement from this SSE on paq8pxd,
    and external ppmd mix didn't help at all, so it seems like paq8pxd actually has
    better mixing and SSE implementations. And paq8pxd mod_ppmd gain on enwik8 was ~100k.

    At least, its clearly visible that I can improve cmix compression by
    mixing ppmd predictions with probability values taken from cmix rc.

    Btw, rc is also an issue - did you see https://encode.su/threads/?p=51672&pp=1 ?

  13. The Following 3 Users Say Thank You to Shelwien For This Useful Post:

    byronknoll (11th March 2017),RamiroCruzo (12th March 2017),xinix (11th March 2017)

  14. #39
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    225
    Thanks
    106
    Thanked 106 Times in 65 Posts
    Quote Originally Posted by Shelwien View Post
    >Btw, rc is also an issue - did you see https://encode.su/threads/?p=51672&pp=1 ?
    I tried computing the cross entropy of cmix predictions (floating point prediction before the arithmetic coding) on several files. This can be used to estimate the file size if the arithmetic coder was perfect. The files output from cmix are only a handful of bytes larger (even for long sequences of 0 bits), so replacing the rangecoder will not have a large effect in practice.

    I made some improvements to lstm in cmix - enwik8 is now 15342703. I am close to being ready for cmix v13 - the last feature I want to investigate is improving SSE. Do you have any advice on how to integrate your SSE code into cmix? Do you think it would be a lot of work? I haven't had time to take a close look at your code - it looks complex. What are the set of files I need to include, and is there a function I can call to pass in a prediction and get back the modified prediction? Thanks!

  15. #40
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,237
    Thanks
    192
    Thanked 968 Times in 501 Posts
    > so replacing the rangecoder will not have a large effect in practice.

    Sure, its just a few bytes in my tests too.
    But its still weird to add redundancy in rc, when the model runs for weeks to reduce it.

    > I made some improvements to lstm in cmix - enwik8 is now 15342703.

    Nice!
    I guess we'd have to restart SSE optimization because of this.

    > I am close to being ready for cmix v13 - the last feature I want to investigate is improving SSE.
    > Do you have any advice on how to integrate your SSE code into cmix?

    Just process the probability value with it, wherever you can fit it, even in rc.

    > Do you think it would be a lot of work?

    Ok, here I simplified it a little:
    1) Include this: http://nishi.dreamhosters.com/u/SSE_model.inc
    2) Allocate an instance of SSE_sh::M_T1 somewhere and call M_Init for it (just S1.M_Init();)
    3) Apply it to a prediction: (note that it takes a probability of 0 in range 1..32767)

    void Encoder::Encode(int bit) {
    const unsigned int p = Discretize(p_->Predict());
    {
    int pp = p/2;
    pp = (pp>0) ? (pp<0x8000) ? pp : 0x8000-1 : 1;
    pp = 0x8000-pp;
    pp = S1.M_Estimate( pp );
    S1.M_Update( bit ); // move to place where the bit is known
    p = (0x8000-pp)*2;
    }
    const unsigned int xmid = x1_ + ((x2_ - x1_) >> 16) * p + (((x2_ - x1_) & 0xffff) * p >> 16);

    > I haven't had time to take a close look at your code - it looks complex.

    Well, the model is automatically generated from optimization profile, so...
    The main component there is interpolated SSE (see sh_SSE1.inc).
    The overall function in this case is like this (mixers and SSE have contexts):
    f(p) = mix1( SSE1(p), mix2(p,SSE2(p)) );

    Or, more precisely, its
    f(p) = X1( mix1( X2(SSE1(X3(p))), X4( mix2( X5(p), X6(SSE2(X7(p))) ) ) ) );
    X# there means static logistic extrapolation, X(p)=sq(st(p)*C), where C is a constant.

    P.S. Btw, please check that its possible to produce the result shown on your site with the windows executable.
    Also, what could be the reason that v12 actually needed >40G of RAM to run, and didn't want to work with swap?

  16. The Following 2 Users Say Thank You to Shelwien For This Useful Post:

    byronknoll (30th March 2017),xinix (1st April 2017)

  17. #41
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    225
    Thanks
    106
    Thanked 106 Times in 65 Posts
    Thanks for the help, I got it working. Running the benchmarks now - I will let you know the results.

    Quote Originally Posted by Shelwien View Post
    >P.S. Btw, please check that its possible to produce the result shown on your site with the windows executable.
    Also, what could be the reason that v12 actually needed >40G of RAM to run, and didn't want to work with swap?
    It is expected for the Windows executable to get different results than the ones posted to my site. I just posted a note about it to my site: it is because the Windows executable was compiled with different compiler options.

    Does cmix actually use >40G memory, or is the issue just that it won't run unless that much is available? I'm not sure what is going on there... it seems to work as expected on my computer (with 32G RAM) in both Windows/Linux.

  18. #42
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    936
    Thanks
    556
    Thanked 375 Times in 280 Posts
    @Shelwien
    > Also, what could be the reason that v12 actually needed >40G of RAM to run, and didn't want to work with swap?

    I've laptop with Win7Pro 64bit, Core i7-4900MQ and 32GB ram and cmix v12 runs properly. Maximum memory usage I saw it was 27GB....


  19. #43
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    225
    Thanks
    106
    Thanked 106 Times in 65 Posts
    Benchmark finished running:

    english.dic: 89697 -> 89582
    enwik6: 180602 -> 180409
    calgary.tar: 549686 -> 549375
    enwik8: 15342703 -> 15330206

    Gains look good! Memory increase is around 425MiB and no significant CPU hit.

    @Shelwien, is this SSE code under the same license as mod_ppmd? Are you the author of the code? I can add some message to the source code to give you attribution.

  20. #44
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,237
    Thanks
    192
    Thanked 968 Times in 501 Posts
    > Benchmark finished running:

    Better than I thought, thanks for testing.
    There's actually a stronger version, which is intended for mixing with mod_ppmd,
    but in fact just takes two inputs.
    Would you be able to test it as is (with two inputs - I can make a similar single-file source for it),
    or maybe its better to integrate mod_ppmd and provide the same interface as previous one?

    > Gains look good! Memory increase is around 425MiB and no significant CPU hit.

    Yeah, I should add memory to optimization metric I think, with SSE its too easy
    to use all the memory - newer profiles seem to use a few GBs actually.
    I mean, it doesn't really need that much - it maybe saved a few bytes at most.

    > Shelwien, is this SSE code under the same license as mod_ppmd?

    Yes, its also available in many of my old projects, for example:
    https://encode.su/threads/1158-SSE2(...ll=1#post22841

    > Are you the author of the code? I can add some message to the source code to give you attribution.

    Yes, sure.

    Anyway, you can expect further improvements once I can get enwik8 probability dump for cmix v13.

  21. The Following User Says Thank You to Shelwien For This Useful Post:

    xinix (1st April 2017)

  22. #45
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    225
    Thanks
    106
    Thanked 106 Times in 65 Posts
    Quote Originally Posted by Shelwien View Post
    Better than I thought, thanks for testing.
    There's actually a stronger version, which is intended for mixing with mod_ppmd,
    but in fact just takes two inputs.
    Would you be able to test it as is (with two inputs - I can make a similar single-file source for it),
    or maybe its better to integrate mod_ppmd and provide the same interface as previous one?

    Sure, I can test out the two input version. Yes, a single-file source would make it easier. No need to integrate mod_ppmd, since that would add a lot of code duplication. I can just create another mod_ppmd instance and add it as the second input. Let me know if you have some recommended parameters for initializing: uint MaxOrder, uint MMAX


    Quote Originally Posted by Shelwien View Post
    Anyway, you can expect further improvements once I can get enwik8 probability dump for cmix v13.

    Great - sounds good. The cmix code in github already has your SSE code enabled, so if you are creating a probability dump it probably makes sense to either disable that SSE code, or use an older revision (c36e893).

  23. #46
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,237
    Thanks
    192
    Thanked 968 Times in 501 Posts
    > Sure, I can test out the two input version. Yes, a single-file source would make it easier.

    http://nishi.dreamhosters.com/u/SSEmix_model.inc
    Same, just M_Estimate has two arguments.

    But you actually have options here. This model is made from 3 models like previous one.
    So you can use the wrapper model, or you can use any of its parts individually.

    > I can just create another mod_ppmd instance and add it as the second input.

    Yes, what I actually meant is that it doesn't specifically has to be a new ppmd model -
    you can use the prediction from existing one, or prediction of any submodel there really.

    > Let me know if you have some recommended parameters for initializing: uint MaxOrder, uint MMAX

    Well, I'm usually testing with o15 m1680, but I doubt that it matters.

    > Great - sounds good. The cmix code in github already has your SSE code enabled,
    > so if you are creating a probability dump it probably makes sense to either disable that SSE code,
    > or use an older revision (c36e893).

    Ok, but I don't have enough memory to run cmix, so I'd have to ask somebody.
    It'd help if you could generate them for me - book1 and enwik8, if possible.
    Previously I did it simply by adding this to rangecoder:
    void Encoder::Encode(int bit) {
    const unsigned int p = Discretize(p_->Predict());

    {
    static FILE* pd = fopen( "book1cmix.pd", "wb" );
    int pp = p/2;
    pp = (pp>0) ? (pp<0x8000) ? pp : 0x8000-1 : 1;
    if( bit==0 ) pp = 0x8000-pp; else pp|=0x8000;
    putc( pp>>8, pd );
    putc( pp, pd );
    }

  24. The Following User Says Thank You to Shelwien For This Useful Post:

    xinix (1st April 2017)

  25. #47
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    936
    Thanks
    556
    Thanked 375 Times in 280 Posts
    Quote Originally Posted by Shelwien View Post
    >
    Ok, but I don't have enough memory to run cmix, so I'd have to ask somebody.
    It'd help if you could generate them for me - book1 and enwik8, if possible.
    [/C]
    I could test something.

  26. #48
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,237
    Thanks
    192
    Thanked 968 Times in 501 Posts
    I guess you can - here I compiled a github snapshot for you, with removed "SSE2" and added probability dumping.
    http://nishi.dreamhosters.com/u/cmix_13pre_pd.exe

  27. The Following User Says Thank You to Shelwien For This Useful Post:

    xinix (1st April 2017)

  28. #49
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    936
    Thanks
    556
    Thanked 375 Times in 280 Posts
    Quote Originally Posted by Shelwien View Post
    I guess you can - here I compiled a github snapshot for you, with removed "SSE2" and added probability dumping.
    http://nishi.dreamhosters.com/u/cmix_13pre_pd.exe
    Ok,Iv'e got it. Which files should be tested?

    Is the book1 file are the same as book1 from Calgary Corpus? - ok, I've foud the answer - YES.
    Where I could download enwik6 file? Should I truncate enwik8?

    I've started to test book1. Using dictionary provides to crash but w/o preprocessing is running. It uses 26GB as far but regarding to compression time it also use swap I think ...

    book1 after cmix13_pd = 185'930 bytes in 2804.14s
    book1cmix.pd file is here: https://www.datafilehost.com/d/e75840ea

    to compare:
    book1 after cmix12 = 186'089 bytes in 839.05s
    book1 after cmix12 with dictionary = 178'062 bytes in 1007.48s (Byron score is 177'967)

    Darek
    Last edited by Darek; 31st March 2017 at 14:33.

  29. #50
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,237
    Thanks
    192
    Thanked 968 Times in 501 Posts
    > Where I could download enwik6 file? Should I truncate enwik8?

    You can - its 1E6 = 1000000 = 0xF4240 first bytes of enwik8.

    > I've started to test book1. Using dictionary provides to crash but w/o
    > preprocessing is running.

    Preprocess it first (with -s), then compress.

    I'd need .pd files for preprocessed inputs - it takes a long time to
    optimize, so I can't do both.

  30. The Following User Says Thank You to Shelwien For This Useful Post:

    xinix (1st April 2017)

  31. #51
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    936
    Thanks
    556
    Thanked 375 Times in 280 Posts
    book1 after cmix13_pd = 185'930 bytes in 2804.14s
    book1cmix.pd file is here: https://www.datafilehost.com/d/e75840ea
    book1 preprocessed after cmix13_pd = 178'216 bytes in 1883.37s
    book1cmix.pd file is here: https://www.datafilehost.com/d/7972b80b

    to compare:
    book1 after cmix12 = 186'089 bytes in 839.05s
    book1 after cmix12 with dictionary = 178'062 bytes in 1007.48s (Byron score is 177'967)

    Darek

  32. #52
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,237
    Thanks
    192
    Thanked 968 Times in 501 Posts
    Uh, 2nd file seems to be wrong? its the same?

    Also, you can test my SSE by putting these files into http://nishi.dreamhosters.com/u/cmix_SSE_v0.rar

  33. The Following User Says Thank You to Shelwien For This Useful Post:

    xinix (1st April 2017)

  34. #53
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    936
    Thanks
    556
    Thanked 375 Times in 280 Posts
    Quote Originally Posted by Shelwien View Post
    Uh, 2nd file seems to be wrong? its the same?

    Also, you can test my SSE by putting these files into http://nishi.dreamhosters.com/u/cmix_SSE_v0.rar
    Hmm, 2nd file was generated during preprocessed book1 compression. It's smaller than first time - w/o preprocessing. Preprocessed file is also smaller = 465'239 bytes.
    The .pd file generated always have the same name "book1cmix.pd" and I wasn't change it.

    One thing - maybe it's important - I've changed name of preprocessed file to biik1 (instead of book1) to avoid the same name files collision and this name file was compressed.

    If it's corrupted then maybe I'll compress it again.

    According to test of SSE - I don't want to doing something wrong then I need to ask how I should use this test properly? Should I just run test.bat?

    @Byron - small question:

    Is should be the same operation if I use

    a) "cmix -c dictionary infile outfile" and
    b) if I preprocess the file using "cmix -s dictionary infile outfile_prep" and then compress it w/o dictionary "cmix -c outfile_prep outfile" ?

    Using "-c dictionary infile outfile" option launches additional "pretraining" stage which takes about 8-10mins and which is not running when I just use "-c infile outfile". This pretraining stage is absent also during just preprocessing option. Why this pretraining stage is used only for option compressing with preprocessing (dictionary) and for pure files not?

    Darek
    Last edited by Darek; 31st March 2017 at 17:40.

  35. #54
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    936
    Thanks
    556
    Thanked 375 Times in 280 Posts
    enwik6 after cmix13_pd = 199'009 bytes in 3630.09s
    enwik6 preprocessed after cmix13_pd = 181'371 bytes in 2286,80s
    both .pd files are here: https://www.datafilehost.com/d/aaa47fe3

    to compare:
    enwik6 after cmix12 = 199'597 bytes in 1101,15s
    enwik6 preprocessed after cmix12 = 181'071 bytes in 1150.78s (Byron score is 181'003)

    Hmmm, In my opinion worse scores of preprocessed files (for book1 and enwik6) for cmix 13 are a results of lack of pretraining stage...

    Darek

  36. #55
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    936
    Thanks
    556
    Thanked 375 Times in 280 Posts
    >Using "-c dictionary infile outfile" option launches additional "pretraining" stage which takes about 8-10mins and which is not running when I just use "-c infile outfile". This pretraining stage is absent also during just preprocessing option. Why this pretraining stage is used only for option compressing with preprocessing (dictionary) and for pure files not?

    Quick test for S.DOC from my testbed:
    141'417 - original file length
    24'388 - cmix v12 with just "-c" option
    23'203 - cmix v12 with "-c english.dic" option - with pretraining stage

    103'921 - cmix v12 S.DOC preprocessed with "-s english.dic" option
    23'248 - cmix v12 S.DOC preprocessed and then compressed with just "-c" option - 0,2% worse score than with pretrained file...

  37. #56
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    225
    Thanks
    106
    Thanked 106 Times in 65 Posts
    Quote Originally Posted by Darek View Post
    @Byron - small question:

    Is should be the same operation if I use

    a) "cmix -c dictionary infile outfile" and
    b) if I preprocess the file using "cmix -s dictionary infile outfile_prep" and then compress it w/o dictionary "cmix -c outfile_prep outfile" ?

    Using "-c dictionary infile outfile" option launches additional "pretraining" stage which takes about 8-10mins and which is not running when I just use "-c infile outfile". This pretraining stage is absent also during just preprocessing option. Why this pretraining stage is used only for option compressing with preprocessing (dictionary) and for pure files not?

    Darek
    Thanks for helping run these tests. Yes, one major difference between those two is the pretraining stage. Pretraining is run on the dictionary itself, which is why it is only run when the dictionary argument is there. One other minor difference: compressing with -s adds a four byte header (recording file size), and then compressing that again will add another four byte header.

    I'm not sure how sensitive Shelwien's optimizations are for these differences. My hope is that it doesn't make a significant difference. It would be nice if the SSE layer doesn't need to be re-tuned for every new cmix version.

  38. The Following User Says Thank You to byronknoll For This Useful Post:

    Darek (31st March 2017)

  39. #57
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,237
    Thanks
    192
    Thanked 968 Times in 501 Posts
    > Hmm, 2nd file was generated during preprocessed book1 compression.

    Well, you somehow managed to set the download links to the same (first) url:
    Code:
    <URL="https://www.datafilehost.com/d/e75840ea">
          https://www.datafilehost.com/d/7972b80b</URL>
    > Using "-c dictionary infile outfile" option launches additional "pretraining" stage

    The problem is that its somehow can't be used on windows? (crashes)
    We had to use the same -s + -c method for v12 as well, even when using original executable.


    > I'm not sure how sensitive Shelwien's optimizations are for these differences.
    > My hope is that it doesn't make a significant difference.
    > It would be nice if the SSE layer doesn't need to be re-tuned for every new cmix version.

    Yes, it should be universal enough.
    But still, the difference between v13 and v12 is significant enough for parameter optimization -
    it should be possible to improve it (enwik8 result) by a few hundred bytes at least.

    Also, the parameter profile that I posted is just not final - its what I made using book1 target,
    in a few days, while after that kampaster was running it for a month with enwik8 target.
    Still need to tweak the coder to take memory size into account and run it more, because of v13
    differences.

  40. The Following User Says Thank You to Shelwien For This Useful Post:

    xinix (1st April 2017)

  41. #58
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    936
    Thanks
    556
    Thanked 375 Times in 280 Posts
    Quote Originally Posted by Shelwien View Post
    >Well, you somehow managed to set the download links to the same (first) url
    My mistake. Sorry for that. This is proper file: https://www.datafilehost.com/d/0322359b


    Quote Originally Posted by Shelwien View Post
    > Using "-c dictionary infile outfile" option launches additional "pretraining" stage

    The problem is that its somehow can't be used on windows? (crashes)
    We had to use the same -s + -c method for v12 as well, even when using original executable.
    The problem is that -s dict + -c method gives worse scores than -c dict and for this executable I can't run -c dict method due to crash then preprocessed scores made by -s dict + -c aren't optimal.

    Generally this version of cmix had quite nice improvements. Examples from my testbed:

    0.WAV
    2'762'044 - original file
    1'395'901 - cmix v10
    1'395'374 - cmix v11
    1'391'800 - cmix v12
    1'375'031 - cmix v13pre_pd

    A.TIF
    2'870'552 - original file
    831'294 - cmix v10
    830'462 - cmix v11
    827'916 - cmix v12
    819'903 - cmix v13pre_pd

    B.TGA
    2'872'818 - original file
    774'368 - cmix v10
    774'158 - cmix v11
    771'799 - cmix v12
    763'005 - cmix v13pre_pd

    Q.WK3
    994'350 - original file
    167'090 - cmix v10
    166'583 - cmix v11
    166'294 - cmix v12
    162'523 - cmix v13pre_pd

    As you could see gain for v13 vs. v12 is bigger than differences between next previous versions...

    Darek

  42. #59
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    225
    Thanks
    106
    Thanked 106 Times in 65 Posts
    Update: I got a nice gain by removing my old SSE code, so now the context mixer feeds directly into the new SSE. enwik8 is 15323969


    I am still running a benchmark with the two input version of SSE (using the existing mod_ppmd model as the second input). So far the results do not look good:


    english.dic: 89580 -> 89580
    enwik6: 180444 -> 180637
    calgary.tar: 549277 -> 549712


    I noticed the memory usage of SSE decreased by about 360MiB, so for some reason it is using much less memory than the previous version.

  43. #60
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,237
    Thanks
    192
    Thanked 968 Times in 501 Posts
    Well, here's what I've got from Darek's "book1cmix_preprocessed" (which is v13 without SSE2) -
    Code:
    465239 - book1
    178215 - book1orig.ari
    178080 - book1s.ari
    177973 - book1sm.ari
    So either you're using it wrong, or v13 has significantly different statistics on these other files.
    Its expected to be used with cmix, which already has mod_ppmd and its original SSE, but not "SSE2".

    Btw, you can test by replacing book1cmix1.pd in http://nishi.dreamhosters.com/u/cmix_SSE_v0.rar

  44. The Following 2 Users Say Thank You to Shelwien For This Useful Post:

    Darek (2nd April 2017),xinix (2nd April 2017)

Page 2 of 3 FirstFirst 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •