I've been surprised at how "in tune" I've become to time, at times, when I've lived outside for substantial periods. Certainly it's not hard to have better than the "faintest idea" when midday or midnight is, if +/- 30 minutes qualifies? Plenty good to decide when to have lunch...

In the higher latitudes, it requires even less acclimation once you've got an idea of your orientation, and maybe what the stars look like. To the point that, at the poles, a clear sky is a direct-reading 24-hour clock.

We sure can judge whether it's before or after sunset though.

Nope. Is sunset when the lower limb of the sun touches the horizon? When it's halfway? When it disappears completely? And is that the actual horizon where the sun happens to be today? The treeline above it? The tower block in the way? Where sea level would be if there wasn't land there? Or just when it gets dark? But then is that civil twilight? Nautical twilight? Astronomical twilight? When the light from the sun gets dimmer than the light from the streetlights that happen to be in your area?