It's usually the source code for the compiler or interpreter we're referring to when we say a language is open source or not. As you say, though, it is a misnomer of sorts. Many languages have both open and closed source compilers.