I first got hooked on data compression in the mid 1990's when the fellow who ran the local video tape store (yes, some of us are that old) who used to build computers for people to supplement his business showed me how amazed he was with the ability of ARJ to compress data better than pkzip (at least I think those were the two compressors if I remember correctly). Anyhow, it got me thinking about data compression and how in the heck this stuff was actually working. I picked up a copy of Mark Nelson's book on Data Compression at the local bookstore that specialized in computer books and I got to reading. At that time BWT was little known and the book largely covered dictionary based compression and statistical compression (LZ and PPM). But I was mostly hooked on the 'magic' of the Burrows Wheeler Transform. Sure I built a few PPM compressors and LZ compressors (and many more professionally as well). But my muse was BWT. At the time I had to write my own BWT algorithm because, frankly, at that time everything sucked. That was the beginning of some of my ideas for MSufSort. The prototype v1.0 would be nothing to write about today but it still the basis for a lot of the methods that make the fastest BWT algorithms even 20 years on. But that was just something that I had to build to get to writing a real BWT compressor which would "change everything". I then wrote M99 and a year later I conceived of what was later published as M03 (published several years after I had conceived of the idea). Did they "change the world"? No. But I wouldn't change how it all unfolded for anything. It's been a hell of a ride.
The point of all of this is this:
Compression, Computer Science, Algorithms etc ... Those of us who are compelled to pursue these muses are truly a rare breed. But we each have to start small. Yes, enjoy the wonder of not quite knowing how it all works for as long as you can, but do not fail to listen to the advice of those who have already walked the path. If you are truly bless with creativity then it will be there for you when you have built the tool set needed to manifest that creativity. First you must crawl, then you can walk.
So rather than trying to start out with the algorithm that will 'change the world', instead, spend a month building the 'algorithm that will enable you to build the skill set to change the world'. Start with a simple LZ compressor. There are still huge improvements introduced in this class even today (just look at Christian's RAZOR). Then move on to tougher challenges from there. The compulsion to pursue these challenges will make themselves known to you, don't worry.
Please don't start with radical claims and no achievements. Instead, start with small achievements and the next thing you know your muse will find you and then you will have the tools necessary to unleash your creativity and give this group something that is truly innovative.
- Michael