You're forgetting about traffic. I've taken the NYC to DC bus quite a few times, and have had anything from 4 to 7 hours in travel time depending on how bad traffic is on the road and whether there's any accidents. Now if they'd put a dedicated bus lane on the highway that'd be much better, but sadly it doesn't exist.

Train service between the two cities, meanwhile, is much more reliable in how long it's actually going to take.