News for hackers. Stuff that matters.
Re:Benchmarking GPT-4 Turbo – A Cautionary Tale by minihat on Thursday November 09, 2023 @09:45AM (#38205621)
GPT-4 Turbo is dramatically worse at one task I often try:

Read the following passage from [new ML article]. Identify their assumptions, and tell me which mathematical operations or procedures they use depend upon these assumptions.

GPT-4: Usually correctly identifies the assumptions, and often quotes the correct mathematics in its reply.

GPT-4 Turbo: Sometimes identifies the assumptions, and is guaranteed to stop trying at that point and then give me a Wikipedia-like summary about the assumptions rather than finish the task. Further prompting will not improve its result.

Re:Benchmarking GPT-4 Turbo – A Cautionary Tale by thorax on Thursday November 09, 2023 @11:07AM (#38206778)
Do you have a link or gist of an example run you tried? I'd be curious to try something similar.