Also the precedent of Concorde is not particularly encouraging: building a supersonic airliner required sacrifices of cabin space and comfort which didn't align well with the preferences of the passengers wealthy enough to afford the fare, so given the choice between paying 5x the subsonic fare for quick but cramped, or 2x-3x for subsonic but luxurious, those passengers chose subsonic.
The amount of discomfort that seems inevitable for the sub-orbital trajectories (i.e., you get to know what it feels like to be an astronaut during launch, shaking, g-forces and all) that seem unavoidable for fast halfway-around-the-world-in-a-couple-hours service probably would sink it economically as anything but an occasional jaunt for rich tourists who want to say they rode it.
(While I never flew Concorde, I did babysit one in a museum. There isn't a single seat that isn't cramped -- even in the cockpit.)