Non-determinism in GPT-4 is caused by Sparse MoE

Re:Non-determinism in GPT-4 is caused by Sparse MoE by mrtranscendence on Saturday August 05, 2023 @12:06AM (#37008920)

> And if these rumors, that GPT-4 is inherently un-deterministic and unreliable, are true then most enterprises are better off finetuning open source LLMs—which are just as capable

Wait, am I misunderstanding you? I feel like I've had a head injury or something, because I've never heard of an open source LLM that's as capable as GPT-4 (in most scenarios).

[ Parent | on HN ]

Re:Non-determinism in GPT-4 is caused by Sparse MoE by visarga on Saturday August 05, 2023 @12:32AM (#37009045)

Only on specific domains, these models don't become generalists like GPT-4, they can become task experts for a single task.

[ Parent | on HN ]

Re:Non-determinism in GPT-4 is caused by Sparse MoE by og_kalu on Saturday August 05, 2023 @08:13AM (#37011246)

Fine-tuned MedPalm is worse than GPT-4 on most Medical Challenge Tests. Fine-tuned Minerva is much worse on arithmetic benchmarks.

The LLM space is just different. There's no guarantee a fine-tuned model will beat a bigger generalist one.

[ Parent | on HN ]