If your visual tool merely exports to ASCII, this won't work. If it stores its data in ASCII, it might. But it needs to be more than just "doesn't use control characters and non-7-bit characters" — it needs to have reasonably short lines that mostly don't reoccur and whose position, if meaningful at all, is meaningful only relative to the position of other nearby things, rather than by absolute line number or byte position or something.
Even then there'd be issues, as e.g. I might want to modify descriptions elsewhere in the file in between modifying the diagram. As a concrete example, I am doing that these days with a spec for a system I'm planning. I'm using a custom Markdown based processor with a filter that takes Graphviz/dot syntax inline, and while I edit the diagram, I'll also then often want to write something about what I've added to it. So if I was going to use an external program, the roundtrip from text editor -> diagram editor -> text editor would need to be very fast and smooth.
Though to be really useful for me, it'd need to work for me via an ssh connection as well...
It's really hard to beat plain text for some of these use-cases. At least without first getting more graphics capabilities back into our terminals.
But doesn't that still apply if you had a visual tool exporting to ASCII? It'd save you the difficulty of moving sections around in a text editor, but give you the visual diff.