Saying that it's mostly cultural IMHO misses the point - having certain days happening in the same season is the whole point of having calendars; unlike the length of weeks and months (which are arbitrary and cultural) there are practical reasons to require an accurate metric for years that doesn't shift seasons around because that affects our agriculture.

For much of human history, farmers needed to know that (for example) this spring is later or earlier than usual, because that affects the length of the growing season they'll see - if they would just start planting solely based on the weather conditions, that's not the best way to go.

Nowadays we could just "directly" make weather predictions based on all kinds of factors, so today we could decouple calendar from seasons if we wanted to, but historically having a calendar that is (for example) two weeks off of traditional seasons would be a bad thing. Sure, you could just learn to adjust, but if you're adjusting anyway, why not adjust the calendar directly - since the vast majority of people were either farmers or derived their income from farming (i.e. landlords / aristocrats), the practical needs of the farming industry outweighed pretty much every other factor.