Amtrak agrees and they've been trying to shut down a lot of those routes for some time now. The problem is that Congress won't allow them to.
Two good targets for this would be Upstate NY, where you could reduce travel time to Buffalo from 8 hours to 5 hours, and make travel to Chicago and Toronto much less painful. Improving the Philadelphia -- Pittsburgh route would also make a lot of sense too.
I'm also not sure what building out NE service does to make Chicago more accessible. Chicago is prohibitively far from all of the east coast cities to make NE rail meaningful.
Would that it were otherwise! I live in Chicago and have lots of friends and business associates in NYC. :|
Similar to what tptacek said below, why should a taxpayer in Arizona support a DC-NYC shuttle?