Look at log4j for a recent example on what unwanted complexity does. For something matching the subject matter at hand here, look at the various image parsing attacks that have gained large notoriety. Now force multiply by having PostScript, an actual programming language, in the path.
Over time, the old format may potentially be removed if it falls in disfavor, or at least only be accepted in more and more restricted contexts, but this works better in closed systems of course. I just finished excising a complex format in favor of a much simpler one, and since it was internal to the organization cutting out the old one did not cause anyone problems. For browsers, I don't know. Deprecating Java Applets and Flash seems to have worked, but those are maybe "heavier" examples than SVG.
As for comparison to existing vector formats, I don't have the necessary domain knowledge there.
Maybe because in the formats named, the image format is the expensive/complex/insecure libraries? I know that sounds a bit snappy, but I felt the author of TinyVG made a pretty good case for what they're aiming at: a simple vector format that does not come with a (or even multiple, in at least the case of PDF) Turing complete kitchen sinks.