IVF babies aren't guaranteed to be infertile, they just have a slightly higher chance of it (because of selection bias in the parents).

My wife and I might well have babies via IVF because she can't get pregnant while on chemotherapy - but that doesn't mean we carry any infertility genes. She just had the bad luck to have some cells mutate into cancer.