Think about it this way: in pre-modern times, fertility rate was high, and women would bear many children, much more than 2 on average. Despite high infant mortality, still much more than 2 would survive childhood. Yet, the population has been growing extremely slowly. This implies that many people haven't had a chance to reproduce, and even among those who could, many of their children didn't, so on average people would have only around 2 descendants who would maintain the progeny line, despite relatively many births -- otherwise the population would have grown, and it didn't. We also have good evidence that it's more intelligent and richer people who enjoyed better reproductive success in pre-modern times, for example from the dying wills of medieval and pre-modern Englishmen. See Gregory Clark's "Farewell to Alms" for references, it's a great book by the way.
Also, the fact that IQ was on the rise was probably due the Flynn effect, which is confounding the issue. There are two effects going on: one is the Flynn effect, which is apparent growth in IQ scores without corresponding increase in what we tend to think of intelligence. The Flynn effect is unrelated to genetic influences on intelligence, it's unrelated to g (which is why it's not predictive of real life outcomes, since it's g that's predictive). At the same time, there was a sort of anti-Flynn effect that I'm talking about above, which is in fact genetic, and acts on g, so it is predictive of real world outcomes. The Flynn effect rise in IQ score was masking the drop in actual intelligence, and now that the Flynn gains are starting to plateu in developed countries, the second effect is starting to become more apparent.
First is that intelligent people tend to obtain more education, and more educated people (especially more educated women) tend to have fewer children on average. This has quite obvious effects from evolutionary perspective.
The second, less obvious, is that evolution works by selecting more fit variants of genes out of the pool, and the pool is made of existing variants and random mutations that happen all the time. These random mutations are typically negative for fitness, so the evolution selects against them. However, humans these days are operating under extremely relaxed selection regime, and gene variants which would have had significant negative impact on fitness in the past, while have the exact same effect today, are no longer as detrimental for reproductive purposes, which is basically the only thing evolution cares about.
For a good reference, see Woodley of Menie & Figueredo, 2013.