That was pretty much the standard in the late 19th and early 20th century. The lines were often economically non-viable and were shut down in the 30s, but on the eve of WWI you had villages of under 1000 people with their own train stations (as small as 20sqm), gauges so narrow you had to walk sideways to fit between the rails (https://en.wikipedia.org/wiki/2_ft_and_600_mm_gauge_railways), and sometimes closer to trams than regular rails with the rails bolted straight onto road surface.

There were even dedicated estate railways!

https://en.wikipedia.org/wiki/British_narrow-gauge_railways#...

Not enough milk in the estate kitchen ? Send a maid with a train to the village to restock!

Also on north Germany they have a 60 cm railway connecting an island at low tide:

https://en.wikipedia.org/wiki/L%C3%BCttmoorsiel-Nordstrandis...

Or the famous Kurobe Gorge Railway in Japan at 762 mm that goes via hard moutain terrain and 40+ (!) tunnels:

https://en.wikipedia.org/wiki/Kurobe_Gorge_Railway

(We travelled it once, can recommend, superb experience!)

Still apparently even if you switch to electric cars and/or trolley buses tire and road surface abrasion is still apparently a significant pollution hazard.

I guess metal wheels on metal rails are less of an issues, so it might make sense to use rail based wehicles more heavily, especially with modern technology improvements (automatic routing/switching, battery powered trains, etc.).