LLaMA can apparently run quantized to 4 bits per param (not sure if worth it though), which would allow you to run a 200TB model on one of these cards if I'm understanding right.
In any case, it's almost certainly not bigger than 1T, even if it's not a dense transformer (PaLM-2 is and makes do with 340B, but it isn't exactly on par).
From the GPTQ paper https://arxiv.org/abs/2210.17323:
"... with negligible accuracy degradation relative to the uncompressed baseline"
A machine like this would top out below 2 trillion parameters using the training algorithms that I'm familiar with.
For LLMs you need to load single row of context size, that's vector of ie. 8k numbers, which is 32kB for single precision floats.
Even for fp64 it adds only 16 bytes.
RMSPRop, Adagrad have half of this overhead.
SGD has no optimizer overhead of course.
But all of that (trainable parameters, activations, optimizer state) is like 12 bytes per trainable parameter, not 80.
For backpropagation you take the diff between actual and expected output and you go backwards to calculate derivate and apply it with optimiser - that's 8 extra bytes for single precision floats per trainable parameter.
Why do you need 80?
The ML engineers in my group chat say "1 DGX to train 7b at full speed, 2 for 13b, 4 for 30b, 8 for 65b"
I hope these models move significantly beyond text at some point. For backend programmers it's ok, but for the rest of the technical world (circuits, mechanical engineering, front end, sound, etc), it's fairly limited.
No, those rumors seemed ridiculous even then. Many AI influencers were posting some of the most absurd material, often makes basic mistakes (like confusing training tokens with parameters), but anyone in the field could have easily told you that 100T parameters sounded ridiculous.
On that note, "100 Terabytes of GPU memory is exactly what you need to train that class of model." is also likely false. That's how much you'd need to fit such a model into memory at 1 byte per param. Not train it.
GPT-4 was trained on image data. Besides gaining understanding of image content it also showed improved language abilities over a GPT-4 trained with only text. Facebook is working on a smaller model with text, image, video, audio, lidar depth, infrared heat, and 6-axis motion data. If a GPT-4 was trained with data like that, what capabilities would it have? Rumor says we will know in a few months.
~6DP * precision
Where D is number of tokens*mini batch size and P is number of parameters.
So if you want to fit fully into memory with a mini batch of 1, context window 32k, and 16 bit precision, that’s 144e12/6/32e3/2 = 375M param.
If you apply one token at a time then
144e12/6/2 = 12 T param
Ofc, in reality you have model parallelism as well…
Hello spurious regression
You can't train a 100T model with "only" 100TB of VRAM, you need for each parameters 4 bytes + 4 bytes (gradient) + 8 bytes (AdamW optimizer) + forward activations that depends on the batch size, sequence length etc, maybe more if you use mixed precision and also you need to distribute the weights.
But in this case what is really ridiculous is the compute requirement. The required compute for optimal model growth roughly quadratically (both your model and your data grow linearly). So for 100T model you need 1e30 FLOPs. This machine gives you 1e18 FLOPs per second. It will take 30k years to train this model on one of these (or 30k of these to train it in a year, but then utilization will start kicking in).
I.E. ; "answer this question where the outcome is the most beneficial to quality of life"
A strange game. The only winning move is not to play. unplugs self
Please ELI5 where I can have a glossary of AI/ML terms - where do I get fluency in speaking about Tokens, Models, Training, Parameters, etc...
Please dont be Snarky - This is info that everyone younger than I am needs as well.
Is there a Canon? Where is it?
https://writings.stephenwolfram.com/2023/02/what-is-chatgpt-...
It's pretty lengthy but doesn't require a PhD to understand. If you can get to the end of it you'll have a much better understanding of what's going on.
That rule of thumb is wrong. The chinchilla paper has it anywhere between 1 and 100 tokens per parameter.
Also, I think training in simulated realities will be big, especially for learning how to interact with complex systems, for developing strategic planning heuristics.