Non-determinism in GPT-4 is caused by Sparse MoE

Re:Non-determinism in GPT-4 is caused by Sparse MoE by albystein on Friday August 04, 2023 @07:41PM (#37007305)

this seems like a plausible outcome, and if true could spell disaster for OpenAI models relative to the competition and open source models. Currently, reliability is one of the core obstacles preventing widespread adoption of LLMs in many business critical workflows. And if these rumors, that GPT-4 is inherently un-deterministic and unreliable, are true then most enterprises are better off finetuning open source LLMs—which are just as capable—for their specific domains. they stand to gain better performance that way anyways, as domain-specific models will always outperform generalist ones

[ Parent | on HN ]

Re:Non-determinism in GPT-4 is caused by Sparse MoE by geysersam on Friday August 04, 2023 @08:16PM (#37007559)

> domain-specific models will always outperform generalist ones

That's only true assuming you habe enough data to train a domain-specific model / expertise to train it and test it correctly.

I've encountered cases where an image recognition task could be accomplished well with a very general model like CLIP, but people still fine-tuned another model on their own small data set because that's considered better.

A domain specific model might be more likely to fail on weird outliers not present in the small domain specific training data.

> could spell disaster for OpenAI

Nah I don't think so. They are not all in on one specific model architecture. If the current architecture is found to have serious unfixable flaws then they'll just change architecture.

[ Parent | on HN ]

Re:Non-determinism in GPT-4 is caused by Sparse MoE by og_kalu on Friday August 04, 2023 @10:18PM (#37008336)

>as domain-specific models will always outperform generalist ones

This is not even close to true for Language models.

[ Parent | on HN ]

Re:Non-determinism in GPT-4 is caused by Sparse MoE by mrtranscendence on Saturday August 05, 2023 @12:06AM (#37008920)

> And if these rumors, that GPT-4 is inherently un-deterministic and unreliable, are true then most enterprises are better off finetuning open source LLMs—which are just as capable

Wait, am I misunderstanding you? I feel like I've had a head injury or something, because I've never heard of an open source LLM that's as capable as GPT-4 (in most scenarios).

[ Parent | on HN ]

Re:Non-determinism in GPT-4 is caused by Sparse MoE by visarga on Saturday August 05, 2023 @12:32AM (#37009045)

Only on specific domains, these models don't become generalists like GPT-4, they can become task experts for a single task.

[ Parent | on HN ]

Re:Non-determinism in GPT-4 is caused by Sparse MoE by og_kalu on Saturday August 05, 2023 @08:13AM (#37011246)

Fine-tuned MedPalm is worse than GPT-4 on most Medical Challenge Tests. Fine-tuned Minerva is much worse on arithmetic benchmarks.

The LLM space is just different. There's no guarantee a fine-tuned model will beat a bigger generalist one.

[ Parent | on HN ]

Re:Non-determinism in GPT-4 is caused by Sparse MoE by og_kalu on Saturday August 05, 2023 @08:13AM (#37011249)

Fine-tuned MedPalm is worse than GPT-4 on most Medical Challenge Tests. Fine-tuned Minerva is much worse on arithmetic benchmarks.

The LLM space is just different. There's no guarantee a fine-tuned model will beat a bigger generalist one.

[ Parent | on HN ]