Well he made that statement in 1965, and then wrote QED, which is a layman's explanation of his Nobel Prize work, in 1985. So perhaps he set out to prove himself wrong.

QED does explain quantum electrodynamics without any calculus or other math (for example, he uses imaginary clocks as a way to explain wave interference), but I think if the average person read it they wouldn't walk away with even a partial understanding of it.