Maybe he was more interested in spending time with their mothers than with the kids... Very few parents prioritize their kids before themselves even if many feel that is what they one is supposed to do.