I think some portion of "genius" is the ability to recognize and run with ideas, some of which come from others. Geniuses not only need to actually have an idea, they need the freedom to pursue it and get recognized.

Getting settled in to a family need not diminish the capacity for ideas, but following the established social conventions of ordinary people will reduce the chances to pursue them. Settling down may also make geniuses less social and as a result they may not pick up as many new ideas from others and they will limit their chances to be recognized.