Integer compression

[LawCounsels]

Input 00 is A , 01 is B , 10 is C , 11 is D ( not 0 10 110 111 )

[Gotty]

What you write does not make sense.

Again:
In your concrete case ("A is 0 B is 10 C is 110 D is 111") how many outputs do you have? How do you interpret them?

A(00) B(01) C(10) D(11) and all exact same probabilities ( 50% , 25%, 12.5% , 12.5% )

Please note that for those probabilities the correct Huffman encoding is "A is 0 B is 10 C is 110 D is 111" as you wrote earlier, and not "A(00) B(01) C(10) D(11)".

If you want to teach your neural network to Huffman encode a file (which is not possible) you'll need to stick to the rules.
Please, first think, then answer.

[LawCounsels]

it's been consistent from beginning in input 1,000 bits '00' is A, 01 is B, 10 is C, 11 is D

We compress with Huffman code table an A (00) in input produces 0 , an input B (01) produces 10.. so forth

[Gotty]

I'm asking about your outputs.
See?

In your concrete case ("A is 0 B is 10 C is 110 D is 111") how many outputs do you have? How do you interpret them?

In your concrete case ("A is 0 B is 10 C is 110 D is 111") how many outputs do you have? How do you interpret them?

So you have "00 01 10 00 11 00 00 01" ("ABCADAAB") on your input (just 16 bits to simplify the case). You want to find the Huffman encoded format, which is "0 10 110 0 111 0 0 10" What is your expected output of the Neural Network?

[LawCounsels]

I'm asking about your outputs.
See?



So you have "00 01 10 00 11 00 00 01" ("ABCADAAB") on your input (just 16 bits to simplify the case). You want to find the Huffman encoded format, which is "0 10 110 0 111 0 0 10" What is your expected output of the Neural Network?

with 16 bits input that will be 8 input symbols ( A B C D), which occurs 4x 2x 1x 1x. Thus output will always be log(base2)[C(8,4,2,1,1)] x 2 number of nodes ( each node corresponds to 1 output bit) ALWAYS

[LawCounsels]

or just ADD 4*1 + 2*2 + 1*3 + 1*3 = 14 output layer nodes ALWAYS

[LawCounsels]

These 14 output bits consists of unique prefixes 0 10 110 or 111, unambiguous interpretable sequential

[Gotty]

We agree on that. See my post. I have 14 bits.
So you will have 14 output nodes giving you some values between 0 and 1. I suppose you will interpret an output as 1 when it is over 0.5 and 0 when it is less. Right?
You train your (general feed-forward neural network) with some cost function. The training needs to be 100% perfect. If it does not produce the expected result even in one of the cases, you are in trouble. If your training set does not contain all the strings you have a risk that the network would not produce the correct, expected result for the missing cases.
So your only chance is to feed all possible input strings to the neural network and train the network to produce 100% results on the output. It can be done with 16 input bits, yes.
Now let's go back to 1000 input bits (1024 to fulfill input requirements). Feed all possible 1024 bit stings to the inputs... Oh... how much time do you have?

[LawCounsels]

We agree on that.
Now let's go back to 1000 input bits (1024 to fulfill input requirements). Feed all possible 1024 bit stings to the inputs... Oh... how much time do you have?

This is trivial you can just initialize with correct crafted sets of weights/bias AND immediate correct for ALL 2^1000 possible inputs SINCE weights here really just exactly do what algorithm does ( 100% correct ALL the Times)

No training required even in this trivial case

[Gotty]

I'm not sure if it's that trivial. Can you provide the weights? Please try it. For the 16-bit case.

Anyway, if you precalculate the weights your whole point becomes invalid. Which is:

relaxes the requirements now allowed to train feed forward NN with target optimum Huffman encoded compressed of training input sets , can it now learn to correct Huffman encode tests new input sets ( not even requires that it now discovered Huffman code ) ?

ie if we now train the feed forward NN with training input binaries data AND comparing to known Huffman encoded compressed output binaries data to back-propagate adjust weights/ bias , can the NN then correct Huffman encode entire new test input binaries data produces correct compressed output binaries henceforth ?

(Boldface = my emphasis)

[LawCounsels]

Yes to be general applicable ( eventual direction) we should just fix some sets of weights so that successive 2 input bits 'hardwired' to produce sequentially 0 10 110 111 ( later eventual direction for training to adapt accommodate various encodings, output layers can now be extra provision ie 3 bits * 500 nodes... 1st '0' delimites Eg 0 or 10 or 110 OR all 111 delimiter. Can generalise this even further later)

Just needs 'hardwired' certain weights so that successive 2 input bits produces sequential codewords ( onto over-provisioned N bits output nodes each)

[LawCounsels]

Concatenate ALL 500 delimited strings 0 or 10 or 110 or 111 from successive 3 output nodes GIVES Huffman encoded compression

[Gotty]

It looks like finding the weights is far from 'trivial'. If you do think it's trivial you will easily post the weights of the 16-bit case.
Nevertheless your question was about neural network training and learning. I hope you already see that it's not really possible to train a neural network to perfectly emulate Huffman encoding.

[LawCounsels]

I am occupied elsewhere for sometime

Let's see if some practitioners already know it's obvious 'perfect' trained with the 'hardwired'

[Bulat Ziganshin]

Of course, once you walked in with genius idea of using NN for XYZ, the rest of task is trivial and can be handled by any experts in XYZ hanging around. We just need to know your name to give you priority in saying that NN can be used for XYZ (jn this case, it's Huffman coding).