And you introduce weird edge cases because days now change while you're eating lunch or driving home.

The phrase "Dinner on Thursday" would become really confusing. ("Do you mean after work on Wednesday? Or before the football game on Friday?")