Sometimes I have to start work at 4am, does that mean that schools should start at 3am to make it convenient for me? What about when I finish at 11pm? What if I'm working overnight?

That sounds incredibly privileged.

https://www.thrillist.com/news/nation/overflow-data-finds-th...

Nearly all workers that perform non-mental labour.

You should really consider talking to some less privileged people sometime.

Most of the world's workers aren't white collar, are they?

Even if they were, are all white collar jobs allowing full remote these days?

Everyone I know that works in a location works shift patterns or other non 9-5 jobs. My wife might start at 7.30 am one day, or another day not finish to 9pm.

If you're working 9-5 every day it's increasingly unlikely you need to be in a specific place to do so. Those who do work in specific places tend to have to cover a far larger range than 9-5 and thus require alternate child care arrangements anyway.