Results 1 to 5 of 5

Thread: Advice to the new - aim high but start small

  1. #1
    Programmer michael maniscalco's Avatar
    Join Date
    Apr 2007
    Location
    Boston, Massachusetts, USA
    Posts
    140
    Thanks
    26
    Thanked 94 Times in 31 Posts

    Advice to the new - aim high but start small

    I first got hooked on data compression in the mid 1990's when the fellow who ran the local video tape store (yes, some of us are that old) who used to build computers for people to supplement his business showed me how amazed he was with the ability of ARJ to compress data better than pkzip (at least I think those were the two compressors if I remember correctly). Anyhow, it got me thinking about data compression and how in the heck this stuff was actually working. I picked up a copy of Mark Nelson's book on Data Compression at the local bookstore that specialized in computer books and I got to reading. At that time BWT was little known and the book largely covered dictionary based compression and statistical compression (LZ and PPM). But I was mostly hooked on the 'magic' of the Burrows Wheeler Transform. Sure I built a few PPM compressors and LZ compressors (and many more professionally as well). But my muse was BWT. At the time I had to write my own BWT algorithm because, frankly, at that time everything sucked. That was the beginning of some of my ideas for MSufSort. The prototype v1.0 would be nothing to write about today but it still the basis for a lot of the methods that make the fastest BWT algorithms even 20 years on. But that was just something that I had to build to get to writing a real BWT compressor which would "change everything". I then wrote M99 and a year later I conceived of what was later published as M03 (published several years after I had conceived of the idea). Did they "change the world"? No. But I wouldn't change how it all unfolded for anything. It's been a hell of a ride.

    The point of all of this is this:

    Compression, Computer Science, Algorithms etc ... Those of us who are compelled to pursue these muses are truly a rare breed. But we each have to start small. Yes, enjoy the wonder of not quite knowing how it all works for as long as you can, but do not fail to listen to the advice of those who have already walked the path. If you are truly bless with creativity then it will be there for you when you have built the tool set needed to manifest that creativity. First you must crawl, then you can walk.

    So rather than trying to start out with the algorithm that will 'change the world', instead, spend a month building the 'algorithm that will enable you to build the skill set to change the world'. Start with a simple LZ compressor. There are still huge improvements introduced in this class even today (just look at Christian's RAZOR). Then move on to tougher challenges from there. The compulsion to pursue these challenges will make themselves known to you, don't worry.

    Please don't start with radical claims and no achievements. Instead, start with small achievements and the next thing you know your muse will find you and then you will have the tools necessary to unleash your creativity and give this group something that is truly innovative.

    - Michael

  2. Thanks (8):

    Gonzalo (23rd August 2019),Hakan Abbas (23rd August 2019),hexagone (23rd August 2019),introspec (23rd August 2019),JamesB (23rd August 2019),Mike (23rd August 2019),schnaader (23rd August 2019),Stephan Busch (23rd August 2019)

  3. #2
    Member
    Join Date
    Aug 2008
    Location
    Planet Earth
    Posts
    1,003
    Thanks
    97
    Thanked 399 Times in 278 Posts
    I agree start small, test and improve, but to get radical new ideas it can be better not first to study a subject to micro improve already existing things, but first try your own thing and then compare it with already existing things. Because once a brain is resonating at a set of populair ideas or rules it's difficult to bend from them.

  4. Thanks (3):

    compgt (23rd August 2019),Jarek (23rd August 2019),rainerzufalldererste (23rd August 2019)

  5. #3
    Member
    Join Date
    Nov 2013
    Location
    Kraków, Poland
    Posts
    791
    Thanks
    240
    Thanked 253 Times in 157 Posts
    While I generally agree with Michael, I prefer Sportsman's view, e.g. there might be no ANS now if I knew AC in 2006.
    Fields are mostly developed by specialists - what is often necessary, but also they "know well what can and should be done" - there is systematic evolution, but it is tough to get out of standard ways of thinking.
    Having a strong base (!), it is worth to attack new problems with clear mind first - before studying and comparing with known approaches.

  6. Thanks:

    compgt (23rd August 2019)

  7. #4
    Member
    Join Date
    Sep 2018
    Location
    Philippines
    Posts
    112
    Thanks
    31
    Thanked 2 Times in 2 Posts
    Aim high:

    "Invent a super compression algorithm and sell it for hundreds of million$."

    You sell it. It's not just: "hey, it improves your credibility to tech companies." It is worth that much already, your deal of a lifetime realized.

  8. #5
    Member
    Join Date
    Sep 2018
    Location
    Philippines
    Posts
    112
    Thanks
    31
    Thanked 2 Times in 2 Posts
    Quote Originally Posted by michael maniscalco View Post
    Then move on to tougher challenges from there. The compulsion to pursue these challenges will make themselves known to you, don't worry.

    Please don't start with radical claims and no achievements. Instead, start with small achievements and the next thing you know your muse will find you and then you will have the tools necessary to unleash your creativity and give this group something that is truly innovative.

    - Michael
    On the other hand, if you're not a good programmer, you don't need to implement all known algorithms. Don't get bitten by the compression coding bug. There should be no compulsion.

Similar Threads

  1. Need some advice on compression artifact
    By boxerab in forum Data Compression
    Replies: 0
    Last Post: 27th May 2014, 20:59
  2. Start command intepretest paramter as file to execute
    By SvenBent in forum Data Compression
    Replies: 3
    Last Post: 15th May 2014, 23:46
  3. Need some advice for order-N CM
    By Sebastian in forum Data Compression
    Replies: 19
    Last Post: 23rd November 2010, 18:54
  4. A high precision timer
    By Matt Mahoney in forum Forum Archive
    Replies: 8
    Last Post: 17th January 2008, 03:36
  5. Start of another BIG and really real Benchmark
    By Simon Berger in forum Forum Archive
    Replies: 31
    Last Post: 15th November 2007, 16:18

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •