I think he's talking about the ~100 generation time-scale; i.e., Darwinian evolution rather than Lamarckian evolution. If anybody has a "desire to have a large family gene", then that gene will come to dominate the human gene pool in a few hundred generations, because people with it will produce, on average, many times as many children as people without it. The two biggest assumptions here are:

1. This gene exists (and is the main factor in determining how many children people have.)

2. Other genes won't become more important.

I would bet on both of these assumptions being false, but it's certainly possible that they hold.