I develop a massively complex healthcare system for running medical practices / small hospitals. We have two support staff members. I think you have it right when you say "a lot of support costs are essentially consequences of software design not being quite right".

There's a number of things that I think help:

* Clear UI

- don't have rows of icons with no text

- try to compartmentalise areas of the application so users can build a 'mental map' of the application (for example we have 'patient home page', 'appointment home page', 'consultation suite', 'accounts home page'. So whole areas of the application work almost like documents.

* Explain everything in clear English

* Make sure the most commonly used actions always 'visible'

* Consider how the user will interact with your product. Healthcare especially is one where you must get this right, an extra click can drive the users crazy. It can be small things that really make a difference, I added a tag-cloud to the top of the medical-record for example, which really helps clinicians when the next patient walks through the door.

* Where possible make good decisions for the user, whether it's pre-selecting options, or removing clutter.

* Don't surprise users - They should never worry about what will happen if they click X

* Don't remove features unless you have properly explained why you're doing it

* Don't change a feature unless you're clearly making it better, or have again got buy-in.

There's probably loads more, but that's just off the top of my head. Then there's the standard stuff like: Never go offline, be fast, etc.

I think also most people would be surprised at how effective 'shiny stuff' is. Making it look pretty goes a long way.

I've been reflecting on this further since reading the initial piece. I wanted to add, just reading this filled me with a sense of calm serenity. This is exactly the opposite of 'firefighter syndrome' where everyone is always heroically fixing things that shouldn't be broken in the first place, and staying late nights that shouldn't be needed. It feels good to be the hero, but you need to stop and work on fire prevention to get to a point where fires almost never happen. If you just play hero and ignore the root problems, you'll end up with a squad of great firefighters that you don't need, and a product that's still on fire regularly.

Unfortunately, it often looks (to others) like the firefighters making these great sacrifices are the best, most dedicated employees (and they may be), but all too often the person who leaves at 5 every day is the one keeping the fires under control in the first place. The most beautiful part of the post was "I am usually done with the email queue by lunch" and browsing social is "kind of the icing on the cake at the end of a busy day.". THOSE are the signs of a good process.

I definitely agree. I think it's really important to have your entire engineering team cycle through support at an interval that makes sense for your company.