One reason for the midnight oil is that the phone stop ringing, everybody else goes home, there are no distractions. Another reason is that the business demands it. Let me give you an example. There is a story that the folks at WordPerfect, which had (and still has, if my mother is any indication) fiercely loyal customers. They had a very family-oriented company culture, and nobody stayed past 5. Microsoft did not go home at five, except maybe at 5 am. Microsoft ate their lunch.
In several situations, I could not get access to computers until after the day shift went home. In Portland, while on co-op during college, I would walk to work at about 0200. The path took me past a bakery that was in full-on baking mode. While building a real-time medical data system, we had one production computer and a second computer that was occasionally used for production at night, so we worked nights to integrate both parts of production so that the other computer would be available for development all day.
And sometimes a problem will grab you and not let you go. The story of fluidics has as its beginning a 36 hour creative burst by its major contributor. And there are problems that I have worked on that have a multi-hour daily mental ramp-up.
The myth part of this is that you can get away with doing this constantly. You can't. As you say this very act should be resorted when you have no choice.
What the midnight oil has taught us in the industry is that the best programming is done in an environment free of interruptions, and that the plentiful availability of development equipment removes scheduling constraints previously imposed on programming.
And I am older than those two numbers and still occasionally pull the midnight oil.
I don't put in 16-hour days, though. There's a clear distinction between the two, and "burning the midnight oil" refers to the former. You can get the advantages of working nights without overworking.
I'm not quite sure about the long-term solution. Normal family life has never been something that many artists are capable of (i.e. Van Gogh), and perhaps companies need to be more accommodating of employees that work two 10-16hrs bursts a week, rather than the normal 9-5.
Personally, I am in the "at risk" category for leaving the industry for this reason -- it is both burn out and lack of interest. I don't know, I still think there is some great code left in me, but it needs more challenging problems then I am facing at present.
But if I've already spent the whole day thinking, then decision fatigue kills my productivity completely. Some might say I'm just tired, but it isn't related to how much sleep I'd had, just how much time I've spent thinking.
I would have been much more comfortable in the cabin but the marina WiFi network didn't reach that far.
Why burn the midnight oil? It was either that or no sailing out next morning to spend a week at sea.
In those days (85-90) version control wasn't a big thing with a 1MB machine, 65MB HD) but my habit of "take a manual snapshot of my current project at the end of every day" is the result of one or two failed "sleepcoding" events.
Tangent: reading the post the other day about how Turbo Pascal fit in 40kB blew my mind and brought back many good memories. I also realized I was much smarter then.
You only need to pull the long hours when a big impact is really needed - generally, learning something new or introducing major new concepts into the codebase. With those projects, slicing up the hours is detrimental to understanding of the problem. You still don't want to make an unnecessarily complicated solution, but long hours make necessary complexity easier to tolerate.
Granted, I'm not writing CompSci research papers on what I do at work, so the difficulty level might have something to do with it.
I mean I was really into this image when I was 20. Now, after had some experience in the field I see that, this very act should be resorted when you have no choice. If you have a tight deadline and all you have to do is code, I mean not think and code but just code - no abstractization is involved, this works. But if you have to plan-code a project this approach sucks. Because you get tired as the night progresses and tired minds do make mistakes. A lot. So while the bugs coalesce you go into smart fixes. Because it has to end tomorrow, right? Then when you return to your code one week/month/year later you'll say: - What the heck have I done here?!
Resting is as important as coding for a coder IMHO. No need to burn ourselves out for a myth. And yes I want to do it when I am 50 or 60.