I think the bigger constraint would be noise. Wikipedia talks about a 70dba noise limit in residential areas after protests and such, but doesn't say what speed exactly they're running at to meet that.

70dba is rather a lot, though, I imagine you'd end up with a lower limit in the US since I don't think most people are going to be easily convinced that another inter-city transit method is going to be worth the noise to them.