Fooling a 9th grader is amazing. That's a pretty well formed human being right there except with less life experience. Fundamentally no different from you in general reasoning terms except on a smaller set of information. So fooling you is merely a question of model size.
It may or may not be fixable without radical redesign. The underlying training objective of mimicking what humans might say may be too at variance with an objective of producing true statements.
I asked it a series of questions about my area of expertise and they were wrong but looked perfectly fine to my wife.
It even confidently “solved” the 2 generals problem with a solution that looks completely plausible if you don’t already know that it won’t work.
All the answers looked good, used several of the correct terms, and one even referenced the project I worked on, but they just contained flat out wrong information.
You reach a point, where when you ask a ML model to generate text given the internet as a corpus, where there just isn't enough text to make something that is both true and convincing. In niche fields, this is just where we are at.
I think you have a point about your tkinter example. That kind of stuff _is_ a lot more convenient than googling and copying and pasting code. But if you push it beyond stuff that you could easily find on stack exchange or in documentation somewhere it doesn't work that well. Like I said, its a search engine with a lot of downsides and some upsides.