I think the sunrise is a better fixture socially, and maybe the sunset is a better fixture biologically. Noon is just an easy to measure compromise for the above two.

Social life and timing mostly revolves around how early you get up. "Its late" kind of means "I am behind schedule to get enough sleep before waking up".

Biologically, we are programmed around sunlight. Whilst our mornings aren't really controlled by the biological clock (that is what the alarm is for), the evening really are. Besides, the for many people, the evening is what separates 'civil life' from 'private life'.