No one is saying these models are conscious or have human awareness of concepts.
It mechanically builds a deeply layered semantic model that correlates to our human understanding.
Quibbling over whether it is "real semantics" or not is just ironically quibbling over semantics. Yes its not conscious, but it doesn't need to be. It is possible to build a mechanical structure that correlates to a human understanding of the world and performs useful tasks that require only mechanical understanding and reasoning, without consciousness or emotions.
So to be precise it mechanically builds a deeply layered syntactic model. LLMs just regurgitate syntax, any semantics can only be imagined by us and overlaid on the syntactic results produced.
If you are right about this, then you should edit or delete this wikipedia article and publish a paper to inform all NLP researchers that there is no such thing as a semantic similarity metric because NLP models cannot understand semantics.
It may internally construct a hierarchy as you set out, but this is and can only be a syntactical hierarchy - though should be no surprise that it corresponds to our usual semantic hierarchy. But whereas our syntax proceeds from our semantics, its syntax proceeds only from our syntax that we've fed it.