From personal experience with a family (two kids) my best time to work on a side project was always between 0am and 2/3am. The only downside is that this doesn't work for a longer period of time as you are getting more and more burned out.

The mornings don't work for me as they start pretty early.