"Language" is a subset of "symbols". I agree with what you said, but it's not representative of the quote in GP.

Just a few days ago was "What do you visualize while programming?", and there's a few of us in the comments that, when programming, think symbolically without language: https://news.ycombinator.com/item?id=41869237