There are research groups (I'm thinking of Tim Dwyer's specifically) that are specifically trying to improve the situation. When you say "it's just a pile of incompatible heuristics", it sounds like you haven't taken the time to try and implement a better solution.
Graph drawing is the kind of thing that seems trivial, right until you try to work on it.
(disclaimer: I'm clearly biased here since I worked with the people who wrote graphviz.)
I hardly think they'd turn away algorithmic contributions, if you've got some.
Do you have an example of your hideous bow ties? I don't usually use graphviz to 'draw' anything, so I don't really have the context to understand your complaint. I've never seen a result I think of as a hideous bow tie.
Page 6 Figure 5.
http://www.graphviz.org/Documentation/TSE93.pdf
This work is absolutely amazing. I've tried implementing it a few years ago (as an excervise when preparing for Google interview). So naive. I've found out that I have read each and every sentence extremely carefully to get things right. I got quite far but never finished the implementation. (And I did not get an offer from Google either.)