Results 1 to 8 of 8

Thread: NNCPa

  1. #1
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    278
    Thanks
    116
    Thanked 160 Times in 117 Posts

    NNCPa

    NNCPa has forked from NNCP 2019-06-29 by Fabrice Bellard.

    Versions of NNCPa:
    2019-11-06 File version 1

    NNCP:
    https://bellard.org/
    https://bellard.org/nncp/
    https://encode.su/threads/3094-NNCP-...eural-Networks

  2. #2
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    278
    Thanks
    116
    Thanked 160 Times in 117 Posts
    NNCPa 2019-11-06.
    Code:
    Option               Description
    cell                 LSTM cell variant (default=lstmc).
                         Added lstmc-f (=lstmc) and lstmc-i.
                         lstmc | lstmc-f = c * f + min(1 - f, i) * in.
                         lstmc-i         = c * min(f, 1 - i) + i * in.
    n_symb n             Number of symbols.
                         Added n=0 (new default): the program automatically sets n_symb with the minimum number of
                         symbols (in the range 0-255) by reading the infile bytes.
    New                  -----------
    proj_size n          Number of projection states.
                         Projection was already implemented in NNCP 2019/06/29, this option has been added to use it.
    block_loop n         Repeats the training on the current block n times.
    block_iter n         Repeats the training specified by block_loop for the first n blocks, then sets block_loop = 1.
    n_embed_hid n        Number of layers in hidden embedding.
    t_embed_hid [0-6]    Type of hidden embedding.
    t_embed_out [0-6]    Type of output embedding.
    ln_embed_hid [0-7]   Layer normalization in hidden embedding.
    ln_embed_out [0-7]   Layer normalization in output embedding.
    ig_activ [0|1]       Input gate activation function.
    fg_activ [0|1]       Forget gate activation function.
    og_activ [0|1]       Output gate activation function.
    ​
    More descriptions are in Changelog.
    Attached Files Attached Files

  3. Thanks:

    Darek (8th November 2019)

  4. #3
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    747
    Thanked 495 Times in 383 Posts
    Scores of my testset compressed by nncpa with best options from RC_1. There some gains and loses. In total 4.7KB.
    However to test more options I'll need to change some parameters to speed up - with these options my testset compress time is about 7 days...
    Do you suggest to test some of the new options as first?
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	nncpa+DBA.jpg 
Views:	37 
Size:	769.0 KB 
ID:	7069  

  5. Thanks (3):

    Hakan Abbas (17th November 2019),lzhuff (17th November 2019),Mauro Vezzosi (17th November 2019)

  6. #4
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    278
    Thanks
    116
    Thanked 160 Times in 117 Posts
    > Scores of my testset compressed by nncpa with best options from RC_1.
    ​In the first version of NNCPa I tried to integrate Shelwien's SSE but I stopped almost immediately, I'll try again in the next versions.
    SSE improves compression but slows down considerably and needs more memory.
    Can you tell me your best current options?
    I used those written here and the results are worse, except for:
    Code:
    284582 D.TGA
    111472 F.JPG
      6280 O.APR
     34751 R.DOC (see below)
     31992 S.DOC
     22085 V.DOC
       422 Y.CFG
    These results can be now slightly different because I don't remember when I made them (maybe I added a few extra bytes to the header).

    > However to test more options I'll need to change some parameters to speed up - with these options my testset compress time is about 7 days...
    Yes, I stopped K.WAD after 2 MB because it takes too long with your options.

    > Do you suggest to test some of the new options as first?
    Try the new weird fg_activ and og_activ options on small files (<~100/150 KB), e.g. (bold: best, underlined: better than w/o -*g_activ):
    Code:
    w/o -*g_activ  -ig_activ  -fg_activ  -og_activ
           111472     111607     111442     111486  F
            44232     311619     101412      75609  J
             6280     154021      71754      11761  O
             4746       4996       4437       4658  P
            34751      35755      34584      33980  R
            31992     740475     117547      51237  S
            22409      23522      22357      22273  T
            10528      11999      10519      10490  U
            22085      23089      21754      21601  V
            16136      16349      15840      15579  W
            12857      13333      12685      12506  X
              422        459        416        425  Y
              249        258        239        250  Z
    Try -block_len.
    If -batch_size = 1, then set -block_len to >= length of the file: you should save 4+ bytes every 1600000 bytes...
    If -batch_size > 1, then set -block_len to -batch_size * 20000-40000, e.g. if -batch_size 3 then -block_len 90000 (3 * 30000).

    Try the new -block_loop and -block_iter.
    I only did few tests, just to start set -block_len to -batch_size * 50000-60000 (regardless of the value of -batch_size), -block_loop 2 and -block_iter 1.

    -*_embed_* can be useful with -n_layer >~5-6, however leave these options aside not because they are less important but because they can have various combinations

    All other new options are less important.

    For example, my best options for R.DOC are:
    -batch_size 1 -n_layer 3 -hidden_size 432 -full_connect 1 -lr 0.0009 -time_step 18 -og_activ 1 -seed 0 -block_len 90000 -block_loop 2 -block_iter 1
    bps=2.203 (33095 bytes)

    When you find a better option, you should re-test all/most the options previously found because they can be modified by the new one.
    Remember that small fluctuations are normal, for example if an option improves from 1000 to 997, this does not mean that it will continue to be better even when other parameters will change.

  7. Thanks:

    Darek (18th November 2019)

  8. #5
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,982
    Thanks
    298
    Thanked 1,309 Times in 745 Posts
    > ​In the first version of NNCPa I tried to integrate Shelwien's SSE but I stopped almost immediately, I'll try again in the next versions.

    That SSE is not really universal, there're many tuned parameters, and overall effect was pretty small, because I couldn't access contexts in actual data.
    The basic component is the interpolated SSE class (sh_SSE1.inc), so maybe you can experiment with that instead?
    (Since it starts with 1:1 mapping and with low wr stays like that; so tuned SSEi never hurts compression).

    Also I wonder if its possible to remove the NNCP's threading method and keep only normal sequential processing of bits.
    It provides best compression anyway (I mean batch_size=1) and for encoding its possible to implement a different kind of MT
    eg. via sorting bits by contexts.

  9. #6
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    747
    Thanked 495 Times in 383 Posts
    >Can you tell me your best current options?
    My best actual options are like in table. Thanks for descriptions, I'll start to test it.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	NNCP_best_options_Nov.jpg 
Views:	39 
Size:	195.9 KB 
ID:	7072  

  10. Thanks:

    Mauro Vezzosi (18th November 2019)

  11. #7
    Member
    Join Date
    Sep 2015
    Location
    Italy
    Posts
    278
    Thanks
    116
    Thanked 160 Times in 117 Posts
    Quote Originally Posted by Shelwien View Post
    Also I wonder if its possible to remove the NNCP's threading method
    I don't know if we can remove it, I guess we can't because it's hidden in the library.
    There is only this in the source files:

    libnc.h:
    typedef struct NCModel NCModel;

    nncp.c:
    static int nb_threads = 1;
    [...]
    NCModel *m;
    [...]
    m = nc_model_init(nb_threads);
    [...]
    case 'T':
    nb_threads = atoi(optarg);

    and NCModel isn't defined anywhere :-( (fields are not used in NNCP *.c/h).

    Quote Originally Posted by Shelwien View Post
    keep only normal sequential processing of bits.
    NNCP process data at the symbol level and I doubt that LibNC works at bit level.
    It is possible to input a sequence of bits by setting -n_symb 2 (the input data will be one bit per byte).

  12. #8
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,982
    Thanks
    298
    Thanked 1,309 Times in 745 Posts
    Yes, but in the end it computes probabilities of bits and encodes bits.
    Just with MT processing of chunks and bits of a symbol - eg. one stream contains all bit15, another all bit14 etc.
    Basically "-batch_size 1" does want I wanted, but its so slow that some other MT implementation would be useful.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •