Exactly this. Supersonic flight is likely to be more expensive than subsonic flight, and will likely mostly be used by people that fly regularly. Even moderately frequent travellers don't find airports that much of an issue because a) their status allows them to skip most of the horrible parts, and b) they have learned how to travel efficiently.