News for hackers. Stuff that matters.

A comment on Mistral 7B, the most powerful language model for its size to date, Apache 2.0. Parent comment

Re:Mistral 7B, the most powerful language model for its size... by HanClinto on Wednesday September 27, 2023 @07:37PM (#37683045)
The source used to create the model is here: https://github.com/mistralai/mistral-src/blob/main/mistral/m...

The source used to train information into the model is elsewhere.

You have the engine. You don't have the source of the data.

[ Parent | on HN ]
Re:Mistral 7B, the most powerful language model for its size... by monocasa on Wednesday September 27, 2023 @09:03PM (#37683845)
That's the inference source as stated in the readme

> This repository contains minimal code to run our 7B model

[ Parent | on HN ]
Re:Mistral 7B, the most powerful language model for its size... by HanClinto on Wednesday September 27, 2023 @09:44PM (#37684147)
The code I linked is the model source -- the structural skeleton. You can create a model with this source and put whatever weights you want in it -- just like Doom.
[ Parent | on HN ]
Re:Mistral 7B, the most powerful language model for its size... by monocasa on Wednesday September 27, 2023 @10:02PM (#37684283)
That is some source. That is not the source behind what built the 14GB model.
[ Parent | on HN ]
Re:Mistral 7B, the most powerful language model for its size... by HanClinto on Thursday September 28, 2023 @04:05PM (#37695045)
If you create a model with this code and save it to disk, it will indeed be 14GB. It will be random weights -- the contents won't be there -- but it will be 14GB.

If you use only what is open-source about Doom, you will have an unplayable game.

If you use only what is open-source about Mistral, you will have an ineffective model.

It's not nothing.

[ Parent | on HN ]
Re:Mistral 7B, the most powerful language model for its size... by monocasa on Thursday September 28, 2023 @07:53PM (#37697563)
The claim is that the complete model, including the weights, is open source. Doom explicitly doesn't cover it's assets as open source.
[ Parent | on HN ]