Pretty much this. LLMs are still limited, perhaps by design but it is so. I recently summarised an article by Maggie Appleton on some of the shortcomings of the model and the implication here: https://twitter.com/kirso_/status/1610849402557722625

disclaimer: I am not the author of the original article.