The accuracy problems you're describing actually support my point: LLMs navigate linguistic structures effectively enough to fool people into thinking they understand, but they can't verify their outputs against reality. That's exactly what you'd expect from a system that only has access to the map (language) and not the territory (reality).
The degree to which LLMs successfully fake understanding depends heavily on how much accuracy you’re looking for. I’ve judged their output as gibberish on a task someone else felt it did quite well. If anything they make it clear how many people just operate on vague associations without any actual understanding of what’s going on.
In terms of map vs territory, LLMs get trained on a host of conflicting information but they don’t synthesize that into uncertainty. Ask one what the average distance between the earth and the moon and you’ll get a number because the form of the response in training data is always a number, look at several websites and you’ll see a bunch of different numbers literally thousands of miles apart which seems odd as we know the actual distance at any moment to well within an inch. Anyway, the inherent method of training is simply incapable of that kind of analysis.
The average lunar distance is approximately 385,000 km https://en.wikipedia.org/wiki/Lunar_distance
The average distance between the Earth and the Moon is 384 400 km (238 855 miles). https://www.rmg.co.uk/stories/space-astronomy/how-far-away-moon
The Moon is approximately 384,000 km (238,600 miles) away from Earth, on average. https://www.britannica.com/science/How-Far-Is-the-Moon-From-Earth
The Moon is an average of 238,855 miles (384,400 km) away. https://spaceplace.nasa.gov/moon-distance/en/
The average distance to the Moon is 382,500 km
https://nasaeclips.arc.nasa.gov/shared_assets/resources/distance-to-the-moon/438170main_GLDistancetotheMoon.pdf
IMO that’s what maturity means in AI systems. Self driving cars aren’t limited by the underlying mechanical complexity, it’s all about the long quest for a system to make reasonably correct decisions hundreds of times a second for years across widely varying regions and weather conditions. Individual cruse missiles on the other hand only needed to operate across a single short and pre-mapped flight in specific conditions, therefore they used visual navigation decades earlier.