It's not completely insane, the part of the brain that gets used for recognising words is very close to the part of the brain that recognises faces. The brain likely cannibalises the part of the cortex that's used to recognise faces to recognise words and letters instead. See this study[1] where the visual word form area reacts much more strongly to faces in illiterates than in people who have learnt to read.

[1] https://www.unicog.org/publications/1-s2.0-S1364661311000738...

https://en.wikipedia.org/wiki/Visual_word_form_area

https://en.wikipedia.org/wiki/Fusiform_face_area