News for hackers. Stuff that matters.

A comment on Mistral 7B, the most powerful language model for its size to date, Apache 2.0. Parent comment

Re:Mistral 7B, the most powerful language model for its size... by monocasa on Wednesday September 27, 2023 @02:49PM (#37679396)
There's absolutely state space iteration in model training. Layer sizes, composition, construction.

> There was no real iteration, you set it up and wait for the GPUs to do the work. The model architecture itself is done before you start training. If you change it, you generally have to start training from scratch.

That's like saying there's no design iteration in software because you type 'make' and the executable is built.

[ Parent | on HN ]
Re:Mistral 7B, the most powerful language model for its size... by spullara on Wednesday September 27, 2023 @07:41PM (#37683089)
[delayed]
[ Parent | on HN ]
Re:Mistral 7B, the most powerful language model for its size... by monocasa on Wednesday September 27, 2023 @09:07PM (#37683879)
Once again, yes, the training cycle is more like a compile step.

Design iteration of the shape of the model absolutely occurs with partially trained weights however.

[ Parent | on HN ]