You've missed a few vital parts of the publishers' role here with your handwave about 'all the stuff that's basically not...writing':
Publishers (the good ones at least) provide editorial support, illustrations, research, design (cover and insides), pr and all the other bits that go into producing a published book (ebook or printed, there isn't actually a huge difference in effort). A book without this is often not a very satisfactory affair. Much as music publishers provide an umbrella under which various services like recording, mixing, production, pr etc are provided which are essential for most musicians to produce their work.
A book at manuscript stage is often inchoate and rough at the edges, and the role of the editor can be critical in suggesting changes or cuts and guiding the author. This is esp. true of non-fiction, but also true of fiction. Just like shopkeepers, publishers are a profession which will probably always exist, as there is a gap between the raw materials and what the public likes to consume, they're not a necessary evil, they're just an intermediary, which is sometimes a useful thing to have.
The publishers have of course taken advantage of the dominance they have over these services to try to wring every last bit of profit out of writers and leave them with the minimum of money for their efforts, and a correction is IMHO due, but there may always be publishers who seek to promote, nurture and improve writers, just as there will probably always be musical publishers/patrons who pay for the ancillary services up-front which musicians can't afford/are not skilled in, but which are essential for producing quality music. That doesn't mean musicians or writers should not have more control and be more fairly recompensed, but there is more to writing and music than is dreamt of in your philosophy.
Before publishing we had the system of patronage, which served a similar function for artists/writers/thinkers but introduced other distortions (influencing what could and could not be written for example), I'm not convinced that was better, but the current system will certainly be disrupted by the web as Atwood implies.
# illustrations
My co-author and me were 100% responsible for all illustrations. I've heard tales of publishers dedicating in-house talent to helping with illustrations, but I do not know how widespread this is.
# research
My publisher was not involved in research whatsoever. I'm not even sure how they could have helped had they wanted to.
# design (cover and insides)
My publisher was indeed responsible for this, but it was up to my co-author and me to decide what class of element each section should contain. As far as I understand most publishers have a house style that the authors have very little (if any) say in.
# pr
This is a huge question mark. There were time when I felt like knocking my head against a wall trying to figure the PR strategy (to misuse the term). I expect some publishers are better than others, but from what I can tell the PR for for programming books is limited to: some website, occasional discount codes, a table at a conference here and there and hooking up interviews with authors on certain websites. As far as I can tell, the weight of a publisher's name is valuable as marketing also, but that seems a bit complacent to me. Any other PR strategies that I've missed?
# and all the other bits that go into producing a published book
I don't know what this means, but I suspect it has something to do with distribution channels.
# provide editorial support
THIS. If I ever write another book the availability of quality editorial support is the deciding factor in my preference for choosing a publisher. The editorial process was the most positive experience in the otherwise arduous book writing process. I understand you can hire third-parties but in going with a publisher they have an editorial staff that can rotate if you find your chosen editor unsatisfactory.
The other points will vary depending on the publisher and the author of course, and some areas (like design and pr) which might seem unimportant, trivial or simply unnecessary to the author might be vital for a successful book. Re PR for example, this encompasses getting reviews in major trade publications/newspapers, getting radio interviews, getting web coverage etc etc, but that'll vary depending on the subject matter - for some books obviously it'd be minimal, so in your experience it might amount to very little added value. Re research, depending on the topic, sometimes editors will do a lot of the work and pull in writers to write specific parts of a non-fiction book - hard to believe if you're a committed, involved author I know. Re design, depending on the book this might be more or less involved - for some books it'll be a simple template where text takes precedence (fiction, programming), for others each page is unique and the illustrations are as important as the text (a children's maths textbook for example).
Other bits was just a catch-all for all the other admin which goes on in publishing a title - buying the ISBN, typesetting, proofreading etc, along with distribution etc, though frankly I think distribution is the least important part of the puzzle, and for many books there simply won't be a dead tree version soon. I suppose my main point was that publishing encompasses far more than the layperson usually considers, and costs are not solely or even mainly associated with printing hard copy books.
So publishers will change with the coming of the internet, perhaps becoming less important, but I suspect they will always be around.
The point is that publishers, in their traditional form, are highly adapted to an environment that no longer exists. At its centre is a reality that printing books is expensive and is cheaper the larger the volume. Someone needs to speculate and invest money upfront to do the printing. Since larger volume is cheaper one big book is better than lots of little books. Risk investors also usually like to minimize investments so that they can know them better. This creates a push for volume and low risk. For that you need to increase upfront spend on things like shelfspace and publicity which then increases the pressure for bigger (higher volume) books. These upfront costs keep eating into (percentage-wise) what is left for the author. Since it means bigger sales, the individual author might be better off but as a whole, the pie doesn't get much bigger. Mostly the pie gets divided in a way that includes fewer authors and more things like financing, screening, etc. Since "publishers" became this amalgamation of services that make up most of the production and distribution price of books, it makes sense to keep adding services. Going from 68% to 71% is not the same as going from 4% - 7%.
Like software, music, video and anything else that got digitized, two things changed: marketing (promotion, sale, distribution) went online and costs went down. Marginal costs to near zero and capital costs to a flexible level (you still need to write the book, but there is no minimum print run). That removes the need for size, removes the need for risk management, makes room for lots of little books, and lots of other things that make the old publisher model somewhat redundant. The different elements get mixed up. Amazon/Apple are sort of lean publishers + retailers + publicists. Thats using the old terms though and they don't really describe the new things very well. Anyway..
Maybe in the future new leaner publishers take over. Maybe authors self publish but hire the services of publicists, editors and such independently. We don't really know.
At this point though it doesn't seem obvious that traditional publishers are any closer to the ideal version than self publishing authors.
There is also a wildcard in here that the product itself can be radically transformed by sales, distribution and consumption mediums. We know about magazines calling short stories into existence. We know about itunes causing a shift from albums to single songs. Maybe short stories will return. Maybe the future is £0.10 per chapter of a 1000 chapter/30 year page book (please nobody tell George RR Martin about this). Maybe its "subscribe to this author. Maybe copyright will die and free will take over.
A friend who is a writer swears blind that his editor make his books massively, massively better as well as doing (along with the rest of the staff at the publishers) a whole bunch of stuff he has no interest in and is no good at.
Yes you can get those services elsewhere but how many authors are going to pay an editor a significant amount up front on the off chance their book sells?
It doesn't always apply - I can see tech books, particularly those written by several people working together, don't rely on editors in the same way - but I think Jeff is generalising from his experience in a very specific field.
I have no evidence of it but it feels to me that publishers get caught up in the whole middlemen are evil, largely derived from record companies and movie studios but publishers, at least not the ones I've spoken to are generally more passionate about what they do, more genuine and authors have far more freedom to move (there are no $100m costs to get a book written) which helps keep them honest.
Edit: That might not be the publishers fault though.. Anyone know how the $29.54 breaks down? How much does amazone take for example?
And yes, writers have to sell a lot of books to make a living. I can imagine a lot of fairly well known writers probably make comparatively little compared to say, a software developer.
It's no co-incidence that you'll often find writers penning newspaper columns or appearing as talking heads on TV shows and the like - it's regular money which supplements the irregular and not massive income they get from writing.
Full breakdown here but the publisher is basically getting about 50% of the list price so your $30 is already down to $15 by the time it gets there:
http://www.askdavetaylor.com/how_do_the_costs_of_publishing_...
He's probably right that tech books are broken but I think it's important to distinguish between technical books and other books which don't share the same model.
For example I doubt R.R Martin has any problems getting a great deal, if he ever wants another publisher for a new series after he is done with game of thrones.
http://www.antipope.org/charlie/blog-static/2009/03/reminder...
(The TL;DR version - because none of the money would go to his publisher and without his publisher his books wouldn't exist.)
What I disagree with is where he says: "almost all the profits go directly to the author. I'm not optimistic this will happen any time soon.", which is interesting. Technically there's a few authors who've been self-made, self-published successes through the Kindle store. You lose a percentage to Amazon so can we class them as a publisher? You could chose to sell an ebook directly to your audience as a few people have tried (and have been successful, there's been a few posted on here I've ended up buying very happily).
Publishers at the moment are a necessary evil. Traditionally they publish your work, handle the distribution and all the stuff that's basically not the writer sitting their ass down and writing. Digitally they seem to have filled a similar role of providing a digital shelf space, dealing with payments, hosting and so on. Until it becomes completely trivially to write it, host it, charge it and deliver it they'll be needed.