"Language models" certainly feels a misnomer. The fact that their inputs and outputs are (actually were, originally) language, doesn't mean that what they model is language. The proof is that there is an infinite number of perfectly correct linguistic productions (think "colorless green ideas sleep furiously") that are not generated by language models exactly because their point is not modelling language but some approximation of the human mind. The fact that the models are trained through language and use language to communicate is purely incidental.