News for hackers. Stuff that matters.
Re:GPT-4o by OutOfHere on Monday May 13, 2024 @02:54PM (#40346864)
I am observing an extremely high rate of text hallucinations with gpt-4o (gpt-4o-2024-05-13) as tested via the API. I advise extreme caution with it. In contrast, I see no such concern with gpt-4-turbo-preview (gpt-4-0125-preview).
Re:GPT-4o by fdb on Monday May 13, 2024 @03:05PM (#40347019)
Same here. I observed it making up functions in d3 (`d3.geoProjectionRaw` and `d3.geoVisible`), in addition to ignoring functions it could have used.
Re:GPT-4o by bigyikes on Monday May 13, 2024 @03:06PM (#40347036)
If true, makes me wonder what kind of regression testing OpenAI does for these models. It can’t be easy to write a unit test for hallucinations.
Re:GPT-4o by OutOfHere on Monday May 13, 2024 @03:20PM (#40347192)
At a high level, ask it to produce a ToC of information about something that you know will exist in the future, but does not yet exist, but also tell it to decline the request if it doesn't verifiably know the answer.
Re:GPT-4o by bigyikes on Monday May 13, 2024 @03:45PM (#40347499)
How do you generalize that for all inputs though?
Re:GPT-4o by OutOfHere on Monday May 13, 2024 @03:58PM (#40347636)
I am not sure I understand the question. I sampled various topics. I used this prompt: https://raw.githubusercontent.com/impredicative/podgenai/mas...

In the prompt, substitute {topic} with something from the near future. As I noted, it behaves correctly for turbo (rejecting the request), and very badly for o (hallucinating nonsense).

Re:GPT-4o by Seattle3503 on Monday May 13, 2024 @11:10PM (#40351302)
I asked it to describe the books of an obscure author. GPT-4o hallucinated books. GPT-4 knew it needed to do an internet search.