Ultimately the costs come down to politics. You can see the problem repeated dozens of times in this discussion--"I'd support X if and only if X was exactly what I wanted". That's not a recipe for compromise, and compromise is fundamental to efficient group decision making.
The costs balloon not because labor is too expensive, but because we take too long to build things and are constantly changing things as we go along. Regulations don't get in the way because they're overwhelmingly difficult or costly; regulations get in the way because the regulatory processes are hijacked and abused to stone wall projects.
We can't get things done in the U.S. because our politics are broken, especially at the national level, but even at the local level.
France keeps extending their network too... Same with other countries.
In the US the last giant scale build-out was permitted before the EPA or Army Corps of Engineers had any regulatory teeth on the matter. Most projects since have been lane additions or incremental extensions. Loops have been built, but are politically easier since they only need to make one metro area happy. There's only been a handful of "new" interstates built since then (at a substantially higher cost). Much of the originally planned interstate system (that was to be a later phase) just never got built.