Why should that not be the case? Those who can write were rich and went to school. It's not hard to imagine that they also received some sort of higher education. The difference between today and 2000 years ago is that everyone is rich by roman standards.
When you watch Clickspring's video series the biggest limitations appear to be access to raw materials and energy. There is a reason why most progress happened in the last 200-300 years. As soon as you have a lathe it is trivial to build every other machine, including the steam machine.
Which on the other hand shows how dangerous it is to have gaps on our knowledge as mankind.
People sometimes think you can't really change IQ via environment but that's because we've worked really hard to eliminate the things that can limit it.
(Plus the Flynn point is that some of the rise is about abstract sophistication rather than actual intelligence rising, people just getting better at what IQ tests test, which would be lower too.)
I'm reading Decline and Fall of the Roman Empire now - and I'm often stuck by imagining how an author 300 years ago managed to write it. It's staggering to think how you could do so much scholarship, writing, and historical analysis without the benefits of computers, the internet, and the modern world. Not to mention the prose, which I absolutely love, and is so dense, thoughtful, and stylish.
I've also read a collection of letters from ancient Rome. Just random letters that have been translated, nobody historical or famous. The writing and thinking in the letters makes it hard to square with the Flynn Effect. The authors seem at least as intelligent, if not more so, than modern people I'm acquainted with.