You don't iterate on the model weights, you train them on billions of tokens. There was no real iteration, you set it up and wait for the GPUs to do the work. The model architecture itself is done before you start training. If you change it, you generally have to start training from scratch.

You do get the ability to further train the model using fine-tuning or LoRas.

There's absolutely state space iteration in model training. Layer sizes, composition, construction.

> There was no real iteration, you set it up and wait for the GPUs to do the work. The model architecture itself is done before you start training. If you change it, you generally have to start training from scratch.

That's like saying there's no design iteration in software because you type 'make' and the executable is built.

[delayed]

Once again, yes, the training cycle is more like a compile step.

Design iteration of the shape of the model absolutely occurs with partially trained weights however.

They trained it from scratch, but there is nothing stopping you from doing some more training on top of it yourself. Nothing is obfuscated, no more information is required beyond the prompting syntax... they just released basically no documentation, which is unfortunately kinda standard in AI land.

There is already a sizable ecosystem of people doing just that: https://huggingface.co/models?sort=modified&search=7B

Just like Microsoft isn't stopping me from patching Windows.

> Nothing is obfuscated

The dataset and pipeline to rebuild these weights are not included.

> The dataset and pipeline to rebuild these weights are not included.

This is a good point.

But its probably not deterministic and reproducible. I bet the researchers themselves couldn't remake the exact weights (or even similar weights) from scratch.

I mean most binaries aren't reproducible. That takes a lot of work and isn't worth it most of the time.

However, I've heard from ML engineers at the big shops that hermetic, reproducible builds are key to making any progress at scale with ML. Apparently that goes all the way back to when Jeff Dean took over Google Brain.

Yep. Even if the initial seed for parameter init, the example shuffling seed, etc were constant, the distributed nature of training (and syncing the model across machines) would kill reproducibility. Not to mention resuming from checkpoints after gradient explosions, etc.

I've heard from ML engineers at larger shops that reproducibility is key to working at scale. That's how you track down "this training regime went to shit because of something we changed" versus "this training regime went to shit because on of the GPUs training it is starting to fail".

Have you tried the HF version which is linked on the main page? It is transformer based after all and it is converted to HF format. Just use QLoRA to fine-tune whatever you want on the top of that model. They handed you hundreds of GPU hours, without asking anything in return. You may throw it away and train the model from scratch. Be ready to throw $70k-$150k into air warming.

> Have you tried the HF version which is linked on the main page? It is transformer based after all and it is converted to HF format. Just use QLoRA to fine-tune whatever you want on the top of that model.

Is that how their engineers built this model in the first place?

Just because they're giving the build artifact of there hard work away doesn't make it open source.

> Be ready to throw $70k-$150k into air warming.

Just because builds are expensive, doesn't mean releasing the build artifacts counts as open source.

I see dozens of your posts here complaining about "not open source." You are either a paid provocateur or a troll. What exactly is missing in your opinion in this release that is making this model not open source? [feeding trolls is fun]

> I see dozens of your posts here complaining about "not open source." You are either a paid provocateur or a troll.

I can't reply in my one tree without being called a troll?

Also, accusing someone of being a paid provocateur is against HN guidelines.

> What exactly is missing in your opinion in this release that is making this model not open source?

The source the engineers at Mistral used to build it.

Is "Doom" open source?

The source code to the engine is available and open, but if you want to do anything useful or recreate the original game, you have to provide your own data.

This model is open source, much in a similar way that the Doom engine is open source.

> Is "Doom" open source?

> The source code to the engine is available and open, but if you want to do anything useful or recreate the original game, you have to provide your own data.

The Doom executable is open source. The data is not. Explicitly, the data is under a different license than the GPL and makes no claims about being open source. There are total conversion mods that replace the closed source levels with open source ones however.

> This model is open source, much in a similar way that the Doom engine is open source.

Here's the source the engineers that created doom used to build the doom engine: https://github.com/id-Software/DOOM

Where is the source used to build this model?

The source used to create the model is here: https://github.com/mistralai/mistral-src/blob/main/mistral/m...

The source used to train information into the model is elsewhere.

You have the engine. You don't have the source of the data.

That's the inference source as stated in the readme

> This repository contains minimal code to run our 7B model

The code I linked is the model source -- the structural skeleton. You can create a model with this source and put whatever weights you want in it -- just like Doom.

That is some source. That is not the source behind what built the 14GB model.

If you create a model with this code and save it to disk, it will indeed be 14GB. It will be random weights -- the contents won't be there -- but it will be 14GB.

If you use only what is open-source about Doom, you will have an unplayable game.

If you use only what is open-source about Mistral, you will have an ineffective model.

It's not nothing.

The claim is that the complete model, including the weights, is open source. Doom explicitly doesn't cover it's assets as open source.

Can you recommend a good resource(s) to learn how to fine-tune one of these on local hardware?

Thanks