Advice to the new - aim high but start small

[michael maniscalco]

I first got hooked on data compression in the mid 1990's when the fellow who ran the local video tape store (yes, some of us are that old) who used to build computers for people to supplement his business showed me how amazed he was with the ability of ARJ to compress data better than pkzip (at least I think those were the two compressors if I remember correctly). Anyhow, it got me thinking about data compression and how in the heck this stuff was actually working. I picked up a copy of Mark Nelson's book on Data Compression at the local bookstore that specialized in computer books and I got to reading. At that time BWT was little known and the book largely covered dictionary based compression and statistical compression (LZ and PPM). But I was mostly hooked on the 'magic' of the Burrows Wheeler Transform. Sure I built a few PPM compressors and LZ compressors (and many more professionally as well). But my muse was BWT. At the time I had to write my own BWT algorithm because, frankly, at that time everything sucked. That was the beginning of some of my ideas for MSufSort. The prototype v1.0 would be nothing to write about today but it still the basis for a lot of the methods that make the fastest BWT algorithms even 20 years on. But that was just something that I had to build to get to writing a real BWT compressor which would "change everything". I then wrote M99 and a year later I conceived of what was later published as M03 (published several years after I had conceived of the idea). Did they "change the world"? No. But I wouldn't change how it all unfolded for anything. It's been a hell of a ride.

The point of all of this is this:

Compression, Computer Science, Algorithms etc ... Those of us who are compelled to pursue these muses are truly a rare breed. But we each have to start small. Yes, enjoy the wonder of not quite knowing how it all works for as long as you can, but do not fail to listen to the advice of those who have already walked the path. If you are truly bless with creativity then it will be there for you when you have built the tool set needed to manifest that creativity. First you must crawl, then you can walk.

So rather than trying to start out with the algorithm that will 'change the world', instead, spend a month building the 'algorithm that will enable you to build the skill set to change the world'. Start with a simple LZ compressor. There are still huge improvements introduced in this class even today (just look at Christian's RAZOR). Then move on to tougher challenges from there. The compulsion to pursue these challenges will make themselves known to you, don't worry.

Please don't start with radical claims and no achievements. Instead, start with small achievements and the next thing you know your muse will find you and then you will have the tools necessary to unleash your creativity and give this group something that is truly innovative.

- Michael

[Sportman]

I agree start small, test and improve, but to get radical new ideas it can be better not first to study a subject to micro improve already existing things, but first try your own thing and then compare it with already existing things. Because once a brain is resonating at a set of populair ideas or rules it's difficult to bend from them.

[Jarek]

While I generally agree with Michael, I prefer Sportsman's view, e.g. there might be no ANS now if I knew AC in 2006.
Fields are mostly developed by specialists - what is often necessary, but also they "know well what can and should be done" - there is systematic evolution, but it is tough to get out of standard ways of thinking.
Having a strong base (!), it is worth to attack new problems with clear mind first - before studying and comparing with known approaches.

[compgt]

Aim high:

"Invent a super compression algorithm and sell it for hundreds of million$."

You sell it. It's not just: "hey, it improves your credibility to tech companies." It is worth that much already, your deal of a lifetime realized.

[compgt]

Then move on to tougher challenges from there. The compulsion to pursue these challenges will make themselves known to you, don't worry.

Please don't start with radical claims and no achievements. Instead, start with small achievements and the next thing you know your muse will find you and then you will have the tools necessary to unleash your creativity and give this group something that is truly innovative.

- Michael

On the other hand, if you're not a good programmer, you don't need to implement all known algorithms. Don't get bitten by the compression coding bug. There should be no compulsion.