Of course it will become easier over time...just like everything. Consider John Carmack and what he did when writing Doom, Wolfenstein, etc...I can almost guarantee you when he was programming, programming was A LOT harder than what it is today. But the difference there is nothing more than time. Even if it's getting easier over time, my main point still stands.
Sure, a simple ray-caster isn't going to need a lot of optimization today. But when you are dealing with the sort of data that is now common. Optimization can be just as important.
While languages have have improved for the better, other things have created other challenges. Consider the PS3 Cell architecture. Considered to be a much more difficult architecture to program for than the processors of Doom's day.
The masters of every subject say stuff like this, and it's really just a type of myopia of the craftsman. I work in manufacturing and people tell you shit like this all the time.
Imagine a 19th Century Car Driver:
"People who think learning to drive should be relatively easy are missing the mastery required for match-rev shifts, air-fuel mixture, spark management - not to mention the intricacies of debugging your carburetor on the side of the road."
Programming will continue to be made more accessible, as is the apparent trend. Nearly no developers know what's really going on under the hood except people who specialize in assembly, CMOS, etc.