jwan584
Re:The Short Case for Nvidia Stock
on Monday January 27, 2025 @03:10AM (#42838614)
The point about using FP32 for training is wrong. Mixed precision (FP16 multiplies, FP32 accumulates) has been use for years – the original paper came out in 2017.
Re:100x defect tolerance: How we solved the yield problem
on Wednesday January 15, 2025 @06:30PM (#42718733)
A good talk on how Cerebras does power & cooling (8min)
https://www.youtube.com/watch?v=wSptSOcO6Vw&ab_channel=Appli...
100x defect tolerance: How we solved the yield problem
Re:Cerebras Inference: AI at Instant Speed
on Tuesday August 27, 2024 @04:19PM (#41372504)
batch size by Q4 will be solid double digits (cerebras employee)
Cerebras CS-3: the fastest and most scalable AI accelerator
Re:GigaGPT: GPT-3 sized models in 565 lines of code
on Monday December 11, 2023 @02:52PM (#38604621)
when you go from 1B to 175B, the model no longer fits in memory. so in practice you have to re-factor the model using tensor/pipeline parallelism. that's why it goes from 600 to 20K LOC.
Re:GigaGPT: GPT-3 sized models in 565 lines of code
on Monday December 11, 2023 @02:50PM (#38604592)
Everyone knows Cerebras by their wafer scale chips. The less understood part is the 12TB of external memory. That's the real reason why large models fit by default and you don't have to chop it up in software ala megatron/deepspeed.
BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model
Re:BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model
on Friday September 22, 2023 @02:35PM (#37615876)
A helpful paper with the full recipe Cerebras uses to train LLMs and their process including:
- Extensively deduplicated dataset (SlimPajama)
- Hyperparameter search using muP
- Variable sequence length training + ALiBi
- Aggressive LR decay
BTLM-3B-8K: 7B Performance in a 3B Parameter Model
Re:Opentensor and Cerebras announce BTLM-3B-8K, a leading 3B param. la...
on Monday July 24, 2023 @03:59PM (#36853427)
Meta announced a partnership with Qualcomm to bring LLMs to mobile. But 3B is a lot more compact than LLaMA's 7B.