This article resonated with me a lot. On Tuesday I wrote a custom build app for my games, something that'll take my game dir filled with crap, preprocess the source code to create multiple versions, filter out unnecessary files, auto generate an inno setup install script and spit out setup files (or run the game in auto play test mode, etc).

Back in high school I wrote a similar thing in C#. I wrote custom code for every window and literally every control - lots of redundant code there. It was a big horrible mess and took two months to develop. For various reasons (cross platform high on that list, but also lack of flexibility and a horrible source base) I decided to rewrite it. I used to think I was a pretty good programmer (Dunning Kruger effect?).

This time around, I instead built a system which automatically generated a GUI around RTTI and handled loading and saving - very similar to Rails' Scaffholding. A day instead of months, because I chose to build a domain specific library with which to construct my application out of.

It's not exactly compiler writing (a parser wasn't needed thanks to RTTI, but if it was I would have written it) but I think it's very analogous to what he's talking about - problems that appear big and nasty can be abstracted down and become trivial by solving one hard problem instead of a larger amount of slightly smaller ones. This is true of compiler writing, frameworks like RoR, layers like HTTP/TCP/IP. It's the very reason why the spread in programmer productivity is so high. Better programmers don't attack problems faster, they attack them from a position of higher leverage.

Finally the dude started blogging again. I am in no position to prove him right or wrong, but his emphasis on parsing/mining skills is very understandable.

Simplifying, commercial software does (mostly) two things: collects data and transforms data. Too many programmers focus on the later ignoring the former. For instance, designing and running cool algorithms on a some sort of a giant graph is cool indeed, but parsing and representing a real-world data in a form of graph is boring and challenging.

"Web 2.0" likes it easy way: "Users will bring you data themselves" :)

That was hilarious. I still can't really gauge how serious that guy was. From the article:

"Situation 7: you have a small, lightweight startup company filled with cool young people with long blue-tinted hair and nose rings and tongue rivets and hip black clothes and iphones and whatever the hell else young people have these days. You use Ruby on Rails for your site..."

lol

I enjoyed this essay, but still didn't get what the big benefit is of learning compilers past the point of creating a parse tree. He seemed to imply that situations 5-7 could benefit from the later part of what compilers do. I can see that you're doing something with the parse tree after you get it, but in each case it looks different from what a compiler would do.

If all that matters is being able to get a parse tree, then maybe more languages should provide a parsing function, just as Lisp does.

I wrote a source highlighter without really knowing compilers, but that's just because getting a Scheme (or BRL) parse tree is trivial.

http://brlewis.com/source.brl?q=/map.brl

In situation 5, I think he's suggesting that being good at making a de-compilers would help a lot in being able to interpret the undocumented router responses. I think maybe he got sidetracked talking about compiler optimization and didn't explicitly mention the sort of practical benefits you'd get from understanding how to do it.

If you are interested in compilers, you might also enjoy Wolfram Research's 2,3 Turing Machine Research Price: http://www.wolframscience.com/prizes/tm23/

Win 25000$ and eternal fame if you can prove that the Turing machine described there is universal, or if you can prove that it is not universal (in the latter case, I think eternal fame won't be included in the price).

It is a beautiful problem.

Well, this is certainly an argument for more simple languages (syntactically) like LISP, C, even Tcl, over C++ and Java. Because I don't think anyone really understands how the C++ compiler really works (it took a looooong time to get a C++ compiler that actually adhered to the standard).

I wouldn't put c++ in the same category as Java. C++ has many layers of "stuff" (templates, macros, etc) while the Java compiler is so simple that you can easily do a reverse-compile.

Personally, I'd rather have a useful syntax and maybe a standard parser that gives easy access to the parse tree. I wrote a Javascript-to-Javascript compiler (that did optimization and correctness checks along the way) once and the fact that JS has syntax wasn't a big problem. (especially since I ripped off the parser from Rhino)

My mistake on Java - frankly, I haven't written any in quite a while.

Lisp has no syntax. :)

Java has much simpler syntax than C, you got your buckets wrong.

Wall. Of. Text.

Seriously, is it necessary?

I enjoy his posts, though apparently they aren't for everyone.

Oh, I enjoyed the post.. I was more commenting about the style of the presentation. There's many ways to break up a wall of text into brain-compilable sections. Joel on Software has an article that mentions one or two. http://joelonsoftware.com/articles/fog0000000033.html

All the writing style and organization in the world is only to get people to read something. If he's gotten you to read it, and you will continue to read what he writes next, he doesn't need to work on all that stuff anymore.

Organization isn't to make the reader's life easier. Making sense of what one reads is largely an introspective activity. No matter how well-organized it is, you the reader have to regurgitate into a form suitable for yourself before you can digest it. This is called active reading.

I don't find "walls of text" at all forbidding, as long as the line width isn't too long. I also find Joel's argument on this point a bit dubious:

When was the last time you noticed a popular magazine or newspaper with entire pages of text?

True, but books do it all the time---as does Joel's essay, except for the gratuitous pull quote in the very section under discussion.

He's just too verbose. If the content took 40% of the words I'd be interested.

But a lot of the other words make it entertaining. Cutting it down to a few power-point slides full of bullets would be a lot more concise, but also boring (and therefore less effective).

Completely agree. Yegge writes a lot, but it's well-paced if you have the time to read it. When he makes a new post I generally set aside some time to read it in its entirety in one sitting.

That's excellent, thanks very much. I'm also reminded of

Je n'ai fait celle-ci plus longue que parce que je n'ai pas eu le loisir de la faire plus courte. -- Blaise Pascal, Lettres provinciales, letter 16, 1657. (I am sorry for the length of my letter, but I had not the time to write a short one.)

Paul, I agree it needs to attract, entertain, and hold the reader but feel the verbosity is laziness in editing rather than style.

I agree it's laziness but I think it's a win for us. If he had to spend weeks instead of hours writing these posts he might not publish anything at all. Those of us willing to wade through his stuff get lots of value for a reasonable cost.

My understanding is that PG spends weeks writing and editing his longer essays. The result is obviously much more solid and digestible, but I'd definitely trade some of that solidity for more quantity. That might be key the difference between a blogger and an essayist.

He doesn't break any of the rules too badly, I don't think. He isn't using large words, or convoluted sentences. He's just saying things perhaps you don't find interesting.

Well, the post is about what's necessary for a programmer to know. So your question "is it necessary?" is answered in Steve's post :)