Great to be hear...... a newbie.

[Hcodec]

Yes, this transform is incredibly flexible. You can place a signal bit in the random stream, or at the end that signals end of block. The block size can be any size you want (to my knowledge this does not exist beyond a size of 2048 for a prp gen because of memory size to store the results in an array). You can manipulate the key so that it filters the permutation outputs. You can insure by key selection that the recursive output will be ascending, descending, odd, even, or repeat after 2^n. There are other methods you can use. Like I have always said. I am not a programmer. I took a B.A.S.I.C. class in college back in 1980, this would be interesting to code.

[Hcodec]

Yes you have to have the original key to decode. Otherwise the output could be any permutation you would have no way to be able to distinguish the decoded output from all permutations of n. An important note when I say all permutation I am not saying N! I am saying N^N * N^N (yes it was supposed to be an encryption method ) why squared is because N = the length of the key, but don't forget this transform is recursive, so it is the length times all the possible times it is looped in a recursive format. Normally you would have to indicate how many times it is looped, but there are obvious ways around that.

For example, lets say we want to compress set Z={2,5,7,2,6,3,3,4,1} we want the output to be highly compressible but we have to indicate not only the key but also how many passes/loops/recursive steps, it took. That information would eat up any gain in compression. So we filter, using a specifc (as talked about above) key and we place an indicator bit in the block so our set now = Z(1)={2,5,7,2,6,3,3,4,1,0} the added ) = end of block or stop here.... do not make any more passes. After we transform this is mapped to (0,0,0,0,0,0,0,0,0,9) after 6 passes, BUT, you now do not need to indicate how many passes it too in the decode just keep making passes until the stop code is reached. Yes, I know the short example is ridiculous because of size but do not forget the block size can be any length, and this easily works with symbols or binary.

What should this transform be called? I've made small attempts at explaining this before and received looks loke i was crazy for even talking about the subject.

[Gotty]

How would an extra code (to signal the end in the input) look like? How would you distinguish it from an actual input number? I think you cannot just have a zero (0) there. The input may contain a zero.

Having a signal bit after every pass with meaning of "stop here" is the same as encoding the number of passes in unary (8 = 00000001, 40 = 00000000000000000000000000000000000000001). That is not a very efficient encoding mechanism. Use some sort of a universal code. They are designed just for that. Still not optimal, because the number of passes has an upper bound and universal codes don't have an upper bound. If you'd know the probability distribution of the number of steps then an entropy encoder (like an arithmetic encoder) would produce the shortest encoding.

[Gotty]

Like I have always said. I am not a programmer. I took a B.A.S.I.C. class in college back in 1980, this would be interesting to code.

No problem. You can always pick it up and learn. You can even do your algorithm in Excel VBA: it is BASIC, and by programming on a spreadsheet you can immediately visualize the result. Anyway you have experience with worksheets. VBA is way easier to learn than C++.

[Gotty]

@compgt
I've never tried, but i would think it would eventually hit a wall because of the signal bits that have to be stored for the decode scheme. Good question, thanks....I'll try it out.

I think this is the best direction for you thoughts. (And thank you compgt for asking the right question. There goes the thank you.)
Hcodec, here you started to feel that not every input is transformable into a lower entropy stream. When you start to apply your encoding on "easy" targets, they may compress well, but as it gets thicker (== more random) you will actually hit a wall. Very true. This wall is in the realm of random sequences
When anyone is dealing with random sequences it is like a very thick wall: you try this way... no compression gain. That way: no compression gain... You really need to try to see and feel it. Hand-picked inputs and keys are not enough.

[Hcodec]

Thanks, it could be this is much better at Encryption than something else.

[Hcodec]

You wrote;

Having a signal bit after every pass with meaning of "stop here" is the same as encoding the number of passes in unary (8 = 00000001, 40 = 00000000000000000000000000000000000000001). That is not a very efficient encoding mechanism. Use some sort of a universal code. They are designed just for that. Still not optimal,

I have a problem explaining things sometimes.... okay most of the time. That is why i usually do not try to talk in the videos. Your signal bit can simply be a single bit added to a block of 128 or 256. If you can change the block to be transformed by a huffman code or be highly compressible list of 1's or 0's you will compress more than you loose by the signal bit. The signal bit is used instead of a number as a stop bit that only take up 1 or 2 bits......i can't see how visual basic can handle this transform. So far i have only explained a few of the beginning steps what it does.

[Gotty]

I lost you.
The purpose of the "signal bit" was to find out (during decoding) how many transformation passes (like 4 or 6 in your earlier examples) were applied on the input to generate the output (during encoding). Right? Or am I completely mistaken? I don't see how 1-2 bits at the end of a 128 (or 256)-bit block would indicate that. I am also unsure what exactly you mean by a "block". It's the output, right? Does it mean that you would encode 128 bits of input to 128 bits of output in this case? (The latter is without the key and any indicator about the number of transformation passes that you applied.)
I suspect that the number of passes would be much higher than 4-6 when the input is much longer. Is this the case? How do you see it?

I haven't received an answer from you about the key selection, yet. (I asked it twice.) It is still blurry for me. Do you just try some and see which one works?

[Gotty]

I have a problem explaining things sometimes.... okay most of the time. That is why i usually do not try to talk in the videos.

Don't worry. Most geniuses have that problem.

[Gotty]

i can't see how visual basic can handle this transform.

All (general purpose) programming languages can do whatever you want. If your transform is not just a theory with undeveloped details, but an algorithm that really works on paper with all the details, then Visual Basic will handle it. Or any other general purpose programming language of your choice. But I suggested VBA for Excel (not VB), because you can program it to read values from spreadsheet cells, and write values to spreadsheet cells. Numbers or "X"-es in your case - exactly what you need.

[Hcodec]

I'll have to at least try. The most i have been able to do is 9 bits for 3 digits, perhaps coding i can see where i can get more.

[Hcodec]

You wrote.

I haven't received an answer from you about the key selection, yet. (I asked it twice.) It is still blurry for me. Do you just try some and see which one works?

Key selection is critical, years ago i started logging which key yielded the response i wanted. Certain keys will produce repetitive cycles others n/2, all permutations in a pseudo random order. Some... if you compare permutation will tell you what key was used even after various passes. Much like aes256 or some hashes like md5. Some keys after a few recursive passes produce very low entropy. There is also an application where you do not use a key but use a block (as in a block, used in cryptography) of the plain text or stream itself.... a while back i found that a revision produces a stream (very difficult to explain) that outputs (easily inversed) the lowest numbers in a frequency. The more the numbers,letters,or symbols are used the lower the number it assigns them. So instead of having to build a tree (Huffman). It is built the same time the file is transformed. A simple simple (yes i know i repeated myself) fast variable length code (i can post a video if interested). So while i have a good handle on the key. I have only attacked the infinate possibilities in spiral notebooks.

[Hcodec]

So far what has been written has been a very basic ideas of what this does. To clarify one of the possibilites of what this is, this transform takes a random stream in base N and transforms to another base N. Let me explain. Original random set of 15 integers in base 10 {9,1,5,3,4,6,7,2,5,8,0,9,6,4,2} after the transform (6,0,2,3,4,5,1,4,5,6,1,2,3,5,6) as you can see this is now in base 7. Through different passes it is possible to reduce this base 7 further, even to base 3, since it is a pseudo random generator, but we will use this example of base 7 for now. If you convert this base 7 output back to base 10 you get 4104294412091 or 42 bits 111011101110011011000000101010111100111011. So you have compressed 915346725809642 (51 bits) to 4104294412091 (42 bits) plus of course the indicator bits of how many passes it took.

Another further explaination about the key. Elements for the key can be groups of symbols, digits, letters. bits, words, nibbles. paths, direction symbols.

[Gotty]

this transform takes a random stream in base N and transforms to another base N. Let me explain. Original random set of 15 integers in base 10 {9,1,5,3,4,6,7,2,5,8,0,9,6,4,2} after the transform (6,0,2,3,4,5,1,4,5,6,1,2,3,5,6) as you can see this is now in base 7. Through different passes it is possible to reduce this base 7 further, even to base 3, since it is a pseudo random generator, but we will use this example of base 7 for now. If you convert this base 7 output back to base 10 you get 4104294412091 or 42 bits 111011101110011011000000101010111100111011. So you have compressed 915346725809642 (51 bits) to 4104294412091 (42 bits) plus of course the indicator bits of how many passes it took.

The problem is the same as with all the above - it will not "compress". When you have random integers, you simply cannot compress them. Any auxiliary information (key, number of passes or anything else) simply eats up the bits and bytes you gained.

Another further explaination about the key. Elements for the key can be groups of symbols, digits, letters. bits, words, nibbles. paths, direction symbols.

I don't follow. But maybe it's not a problem.

An idea remains just an idea until you try to implement it. When implementing you either prove that it works and can compress random integers/digits into a shorter representation (with all the information required to reverse the process i.e. to uncompress) or you will meet the beauty (or horror) of randomness.

[Hcodec]

The problem is the same as with all the above - it will not "compress". When you have random integers, you simply cannot compress them.

I am not seeing the problem that you are... very likely i need glasses. Most all (since i have not tried all even though i have been doing this for years) 10 digit integers can be transformed into base 4 or less in 50 passes or less. Many times down to base 2 which means 1 byte 2 bits to represent a 10 digit integer. What am i missing when random numbers can be transformed into a stream with more sequences than words? Btw...the same key is used for the entire file. You do not keep changing keys every 10 digits or every pass.

[Gotty]

Please try it in practice. Without that it's just an idea.

[Hcodec]

Agreed!