I don't claim to have the answers, but a few years ago, I had to help out some EE's write some testing software in LabVIEW. The first thing that shocked me was the system was immune to bad data. Secondly, how elegantly it took advantage of our multiprocessing and multithreading hardware.
What I wish systems researchers would work on is concurrent, signal-based, synchronous languages. This is a winnable game, but we must swallow the red pill and change the rules of the game with new tools and fresh ideas.
The example of FDs from the original article seems like a good one; a properly developed expert system might have been able to notice a lack of changes in the current software environment and suggest looking elsewhere. (Problem is, if I got that response from my automated problem-solver I'd just ignore it and keep looking . . .)
So, any insightful points to crush my youthful idealism? :)
First problem: Need some 'experts'. So, only know about problems the experts do. Their knowledge is supposed to be mostly just empirical, from their 'expert experience'. Even if they have 'deep knowledge', that is, how the systems work internally (e.g., the engine connects to the torque converter connects to the transmission connects to the rear differential connects to the rear wheels), the specific problems they know about are usually just the ones they have encountered. So, essentially can only address problems seen before and are well understood.
But for the problems of the OP, they were seen for the first time. Bummer. Actually, with irony, if the problems have been seen before and are well understood, then why the heck have they not already been solved?
Second, with expert systems, we're working just intuitively from just experience. So, we have little idea if what we are doing is the best possible in any reasonable sense or even at all good. In particular, in monitoring, the main, first operational problem on the 'bridge' or in the 'NOC' (network operations center) is the false alarm rate too high with no good way to lower it (except just to ignore some alarms, but we should be able to do much better).
Net, for a serious attack on system monitoring and management, I can't recommend taking expert systems very seriously.
We passed the point quite a while back where "applied CS" -- what it's like to build and maintain scalable systems in the real world -- looks a lot more like anthropology than like mathematics.
We have tools like sampling profilers, loggers that do stateful packet inspection on a small percentage of traffic, and sophisticated A/B testing frameworks that reason about user populations rather than individuals. But I think we could probably build a whole 'nother generation of these tools by thinking about this problem of understanding complex systems from some new perspectives. I nominate anthropology as a discipline that has many years under its belt wrestling with the impossibility of complete description. (And wrestling with other tough issues, sometimes not well, but that's another conversation.)
I'm not talking about the kind of "anthropology" work that companies like Intel have done for a long time, which are firmly in the social sciences tradition and are really user studies. (Though that work is valuable.) I'm talking about porting ideas such as "thick description" to our technical toolkit.
http://en.wikipedia.org/wiki/Thick_description
They're chaotic systems[1]. Parameters which are below the threshold of detection can drastically affect their trajectory in phase space. Accidental complexity has zoomed off to live a life all its own, and that takeoff has been continuous for decades [2].
Everything we build around software that's not directly bearing on the problem domain is chipping away at this or that aspect of the ungoverned bits. As we corral each level of new unpredictability, we merely set the scene of the next level of complexity.
There is no universal solution. There is no final boss which, once defeated, leads to the scrolling names and 8-bit themesong. It never stops, ever, because the value of systems that frequently fail unpredictably is lower than the cost of flawless systems and higher than the value of not building such systems.
Edit: sounds very unhelpful, even defeatist. I know. I'm deeply ambivalent about what can be done versus what we should assume is possible. I'm a fan of serious software engineering and tool support and all that jazz ... yet thoroughly pessimistic about the limits of the possible. Deep sighs for every occasion!
[1] http://chester.id.au/2012/06/27/a-not-sobrief-aside-on-reign... (warning, self linkage)
[2] http://www.cs.nott.ac.uk/~cah/G51ISS/Documents/NoSilverBulle...