News for hackers. Stuff that matters.
Re:Large Enough – Mistral AI by nathanasmith on Wednesday July 24, 2024 @12:17PM (#41058595)
They also said in the paper that 405B was only trained to "compute-optimal" unlike the smaller models that were trained well past that point indicating the larger model still had some runway so had they continued it would have kept getting stronger.
Re:Large Enough – Mistral AI by moffkalast on Wednesday July 24, 2024 @12:52PM (#41059038)
Makes sense right? Otherwise why make a model so large that nobody can conceivably run it if not to optimize for performance on a limited dataset/compute? It was always a distillation source model, not a production one.