News for hackers. Stuff that matters.

A comment on Notes on OpenAI's new o1 chain-of-thought models. Parent comment

Re:Notes on OpenAI's new o1 chain-of-thought models by awb on Friday September 13, 2024 @11:12AM (#41531968)
It probably depends on your problem space. In creative writing, I wonder if its even perceptible if the LLM is creating content at the boundaries of its knowledge base. But for programming or other falsifiable (and rapidly changing) disciplines it is noticeable and a problem.

Maybe some evaluation of the sample size would be helpful? If the LLM has less than X samples of an input word or phrase it could include a cautionary note in its output, or even respond with some variant of “I don’t know”.

[ Parent | on HN ]
Re:Notes on OpenAI's new o1 chain-of-thought models by 0xdeadbeefbabe on Friday September 13, 2024 @11:31AM (#41532145)
> It probably depends on your problem space

Makes me wonder if the medical doctors can ever blame the LLM over other factors for killing their patients.

[ Parent | on HN ]
Re:Notes on OpenAI's new o1 chain-of-thought models by freejazz on Friday September 13, 2024 @02:54PM (#41534017)
> I wonder if its even perceptible if the LLM is creating content at the boundaries of its knowledge base

The problem space in creative writing is well beyond the problem space for programming or other "falsifiable disciplines".

[ Parent | on HN ]
Re:Notes on OpenAI's new o1 chain-of-thought models by ijk on Friday September 13, 2024 @03:48PM (#41534514)
In creative writing the problem becomes things like word choice and implications that have unexpected deviations from its expectations.

It can get really obvious when it's repeatedly using clichés. Both in repeated phrases and in trying to give every story the same ending.

[ Parent | on HN ]