Or skip the middle man and let people keep their dollars and spend them how they want?

How do you decide on the path the road should be built on? Everybody building a few feet individually?

Most people are not debating the building of roads, and most of those are at state and local levels anyways. The bigger debate is on the actual federal budget, which has ballooned over time into all sorts of items which cannot be assumed to be uncontroversial or even generally agreed on by a majority of citizens.

A lot of us would prefer if the federal government actually did allocate some more of its funds to maintaining the roads and infrastructure that we have; that would be a great start.

They probably agree that A road has to be built, but exactly WHERE gives you about as many opinions as 5hwre are participants in the discussion. This requires some approach for structuring the decision making. Representative democracy is one. And yes, for such hands on topics it is relatively simple. But "big topics" have equal problem with many opinions, where some form of consensus has to be found.