Today California High Speed Rail and Texas Central and Florida's Brightline have to deal with angry suburban neighbors, angry farmers, and legal and environmental review processes that are the result of a century and a half's worth of hindsight.
There really isn't any "environmental impact" for construction within the city (the environment there is already destroyed). But that sure does provide fodder for endless delays and expense.
> 20MPH
Travel from Omaha to San Francisco was 4 days at the opening, including stops. The distance was 1,912 miles. So the average was indeed 20, but to achieve that average (including stops) it must have been going at least 40 on much of the route.
Quite a difference from the alternative, months of walking.
If you like, the first Boston subway was completed in 1897, it took 2 years, and it was all done with muscle, too.
Seattle is a bit of an odd case when talking about construction timelines, because the projects are drawn out mostly due to a state law that limits how much Sound Transit can bond out at any given time. Anything more aggressive than that would require passing a 60% vote threshold, which would be quite the landslide election. (It's also how the first Seattle subway attempt died.)
Environmental impact doesn't just concern spoilage of a pristine environment; among other things, it can consider impacts on drainage and runoff, noise pollution, construction impacts, efficacy at diverting car trips and CO2 emissions, etc.
Actual construction, even nowadays, is pretty fast. What takes 20-40 years is the long-term planning. By comparison, the first intercontinental railroad was advocated first in 1832 (before most of the American West was even American), and serious planning first started in the late 1840s and early 1850s, which led to the Gadsden Purchase in 1853.
For comparison, the first intercontinental railroad was built by hand and muscle in 6 years. Including the bridges, cuts, and grading. Including through a mountain range.