That's not the case. Children can learn how to drive in snow or dirt roads in a matter of hours. Learning to drive on public roads has more to do with learning the rules than learning to drive a car per se.

This. Lots of kids on farms learn to drive the farm truck off road as soon as their feet can touch the pedals while seeing over the steering wheel.

Any folks who grew up on a farm who can confirm?

I didn't grow up on a farm, but my grandparents were from rural Arkansas and raised me in a small Texas town. I had plenty of opportunity to learn to drive cars on back roads and had my own dirt bikes and motorcycles by age nine. (Modern day helicopter parents would boggle at the freedom we kids had in the 70's). By the time I was twelve my grandmother retired and bought a 3.5 acre plot in rural Texas.

I wasn't allowed to fell the trees but I had to use a chainsaw and cut the branches off the felled trees and then section the denuded trees into small enough rounds they were suitable for busting into firewood with an axe. I was given the option to learn to drive my grandfather's stick shift Toyota pickup truck in order to move all the wood with that, or move it by hand. It was an easy decision.

Driving a tracktor with it's big nobbly wheels and low gear ratios along a dirt path is trivial compared to trying to coax your Honda with worn out tires up a icy hill or out of the mud. The two just don't compare.

I learned to drive a small manual shift tractor at 7 and mowed a 4 acre lawn with lots of trees and bushes. Also snowmobile and motor boat.

Learning the rules and developing situational awareness.

But, yes, until you're talking about more serious off-road/bad road (which is challenging in a completely different way), there's nothing especially hard about driving on a private dirt road.

I did my driving education during the summer and got my license. Got my first car during the middle of winter and just learned to drive in snow (we did have a 2 hour course on a slippery track as part of the driving education). It isn't really hard. Just have proper winter tires and don't go too fast.

Also driving the shitty back roads in Finland was a lot of fun in my old Honda Civic which didn't have power steering or ABS (let alone any kind of traction control and stabilization you have in newer cars). Driving a modern car with all the modern aids is very easy on dirt. On snow some of them can hinder you though (some automatic gearboxes just don't know what to do when none of the wheels have any traction) but for the most part it is very easy too if you have winter tires.

Driving is very easy. Too easy I think which causes people to drive way too fast for the situation quite often.

Main issue for driving on dirt roads/snow for automated cars I see is just seeing the road itself correctly not with controlling the car.

I don't think you're wrong, but I predict that this will be a complete non-problem: by the time people's skills have atrophied significantly, self-driving car technology will have improved to the point where it won't matter anymore.