Well yes, ChatGPT is still not a search engine.

But it's weird how for certain questions it says it's not an expert in that field, or simply that it doesn't know what you're asking for, yet sometimes it goes ahead and isn't just slightly wrong but makes up a lot of BS. Like,I asked it about a band's members and it made up three names. When I said that it's wrong it apologized and made up three different names. I googled the names and nothing came up. Then I asked for some songs by that band and I got more completely made up stuff that wasn't even titles of any band according to more Google and YouTube searches.

It appears to me that it is a big problem of the training set containing mostly questions that can be (and are) answered. Sure, on StackOverflow you will have questions where the response is: please refine your question. But even then, did they use the raw input, or did they use the edited input? Similar with interviews: those are not actual interviews (for the most part) but edited for clarity. Hence, I doubt that GPT was much trained on reiterating questions and instead learned that there is always an answer, even if there is none to that particular question or when it does not know.

Then we may have just found an important philosophical concept.

I'm an accountant, and when someone asks me about the rash on their skin I say, 'I'm not a doctor, I'm an accountant'.

When you ask ChatGPT something that is not it's job it just bullshits its way through like a cheap salesman.

I don't know I assume it would tell you put some cortisone cream on it or to seek a dermatologist. 99% of internet advice boils down to that I don't think chatgpt/gpt 4 would be much different in that regard. It definitely can bullshit but for really generalized stuff like that I doubt it.

It's not like your asking a singular person the question like it is when you ask an accountant but an abstraction of written human thought and interactions.

It is though. I asked it today what, in Python, advantages/disadvantages of dataclasses vs. dictionaries vs. global variables are. Perfect answer right away, with actionable recommendations. Using a search engine would have cost me at least 15 min to figure out what works best in my use case.

Sure, but it acts as if (and should have) the ability of someone unreasonably good at pub trivia and jeopardy. Its not just questions deep in the weeds about specific companies and product lines where it falls flat. I've seen the same error mode in large city transit questions. Ask it which trains go to a particular station, it will list more lines than actually do and also forget a few. Ask it to list which stations one of those services stops at, it will not have the original station you asked about in that services listing. Forget knowledge, it doesn't even have consistency.

It seems to me that it has a problem with questions about incidence structures, scenarios where objects of type X are just lists of incident objects from set Y, and objects from set Y are just lists of incident objects from set X. A station is just a list of all the services which go there, a service is just a list of all the stations it goes to. A product line is just a list of products, each product is part of some product lines. A geometrical line is just a co-linear set of points, a geometrical point is just the intersection of several lines.

Conjecture: any time you ask it questions which involve reasoning about an incidence structure, it will either hallucinate incidences that aren't or neglect incidences that are. The logic of incidence is lost on it, the answers it gives are inconsistent and contradictory.

Yes, really weird fabrications come from that thing.

I asked to to summarize an online article, and it fabricated an author's name, despite the line "Author: [author name and firm]" being right under the top.

I then challenged it, and it apologized for the confusion and error, then fabricated a different author's name. And a wrong date of publication, which was also listed. It eventually got the date right, but continued fabricating names until I gave up laughing. Even after I told it to look on the line beginning with "Author". Odd thing was, it threw an error when I told it about "Author: FName", as if it freaked out when asked to tell a fact. But then it just fabricated another with "FName Fabrication#8"...

Actively, cheerfully, malicious...