This isn’t about “grown adults” and their capabilities. It’s about the well-being of society as a whole. There’s a lot more to consider than “when should working people show up for work”?

Note that a significant portion of society doesn’t have a day job — kids, retired/elderly, caregivers & home-parents, disabled and unable to work, etc. In fact, the needs of these groups ought to be prioritized, since as you said, working adults can adapt.