And Chicago/NYC is the "easy" connection. Atlanta, Texas Triangle, and Florida are simply too far away, let alone trying to reach across the Plains to hit Denver, Phoenix, and Las Vegas en route to LA/SF.
By my count, a NYC-EWR-PHL-HAR-PIT-CLE-TOL-SBN-CHI train would directly serve metros with ~50 million people (and indirectly serve much of the rest of the northeast and midwest).
Sure, you can fix issues with tunnels and bridges. But that doesn't come cheap. And what is by far the most expensive portion of the line is the part that's necessary for any sort of traction, which greatly increases the chances of the entire project being a white elephant (keeping in mind, again, that even in the best-case scenario, you're still at the very margin of being able to run a profit). The economics look worse than the Turin-Lyon HSR, which is already criticized as an expensive, unnecessary boondoggle.
In other words, the 5 hour rule of thumb predicts that a modern system where the coastal cities were densely connected and there were a few money-losing lines that crossed unpopulated regions would be wildly profitable.
State-of-the-art high speed rail (like in China) would be even faster, and make even more sense.