Idea - Don't flame me pls...

[rubendodge]

Okay now I am not sure if this is a good idea or not but what if you were to just xor all the letters one at a time (each letter having a specific XOR making all the letters equal the same thing) to make them all the same letters and then do as follows like bwt does example (award=aaaaa=a4). That way all characters that are printable can be compressed (besides spaces and new lines to preserve the file format) would this be a possible text compression technic? I know I am not a advanced enough programmer to do this yet but I will make it my next project to atleast try. Anyways tell me what you think?

And example could be seen here http://guymal.com/mycode/xor_js_encryption/ taking the required xor for each letter making all the letters the same yet still maintaining file format?

Also note the decryption part of the program would hold the specific xor's used to undo it all and restore the file?

P.S. You neccasarily dont have to xor each entry you could always implement switch statements for each letter and then increment the letter till its the specific one.

[rubendodge]

Okay I thought about it and finally found some very good reasonw hy this idea wouldnt work. Who knows maybe someone who knows more about this stuff can find a way to make it work and can develop a new improved text compression algorithm lol.

[Shelwien]

There's a common technique called "dictionary preprocessing" that does something similar.
Also some comparisons take into account the decompressor size
(thus sizes of sfx archives are effectively compared) and in such
cases the possible gain from such methods is very limited.
Well, you always can "compress" the file into its ed2k hash or something.

[nanoflooder]

Yeah but there will be one or two issues with decompressing it back

[Shelwien]

Why, you can't really do anything anymore without internet access.
Just give your archive an attactive-looking name, and it would be shared
on p2p forever...

[Black_Fox]

I wouldn't be so sure with current avalanche of TPB-process-related threatening to torrent portals.
As to comment to the idea, this would change nothing (original file changed into A's and then database containing original characters, effectively equal to the original file).

[encode]

Yep, the encoder should accurately predict the next character to xor with. If we predict ALL characters correctly we may output a file with zeroes - thus indicating the number of input characters. In particular way, we may compress file into a few bytes - indicating the input size with one integer... As example, if we compress the pseudo random number generator output and we KNOW the algorithm, we may do such thing here.
Yet another crazy idea is to use "the predictor" algorithm idea, but instead of dropping flags - predicted/mispredicted+char we may just xor with a guessed char. Anyway, I tested the idea, and in practice this not helps...

[rubendodge]

wow oh well it seemed like a good idea at the time but I guess the other methods you guys use would definatly be more effective and less problematic. Anyways btw encode I like ur site.