At some point, the models _have_ to do "continuous integration" to provide the "AGI" that's wanted out of this tech.
https://mlnotes.substack.com/p/the-valleys-going-crazy-how-d...
I feel like a fraud for telling people that DeepSeek could've smuggled chips once I took a look at the actual DeepSeek optimizations. The only reason why a team would allocate time on memory optimizations and writing NVPTX code rather than becoming much better at posttraining is if they severely struggled with memory during training.
I mean, take a look at the numbers:
https://www.fibermall.com/blog/nvidia-ai-chip.htm#A100_vs_A8...
This is a massive trick pulled by Jensen, take your best chip that's regulated by the government, find which specs are required for the regulations and then make a weaker clone that bypasses one of the specs.
Tech or politics related, he's off the deep end.
He's a lucky mensch, no more, no less.
I can run the largest model at 4 tokens per second on a 64GB card. Smaller models are _faster_ than Phi-4.
I've just switched to it for my local inference.
From DeepSeek-R1 paper:
> As shown in Table 5, simply distilling DeepSeek-R1’s outputs enables the efficient DeepSeekR1-7B (i.e., DeepSeek-R1-Distill-Qwen-7B, abbreviated similarly below) to outperform nonreasoning models like GPT-4o-0513 across the board.
and
> DeepSeek-R1-14B surpasses QwQ-32BPreview on all evaluation metrics, while DeepSeek-R1-32B and DeepSeek-R1-70B significantly exceed o1-mini on most benchmarks.
and
> These [Distilled Model Evaluation] results demonstrate the strong potential of distillation. Additionally, we found that applying RL to these distilled models yields significant further gains. We believe this warrants further exploration and therefore present only the results of the simple SFT-distilled models here.
I run my models in an agentic framework with fast models that can ask slower models or APIs when needed. It works perfectly, 60 percent of the time lol.
Did training and inference just get 40x more efficient, or just training? They trained a model with impressive outputs on a limited number of GPUs, but DeepSeek is still a big model that requires a lot of resources to run. Moreover, which costs more, training a model once or using it for inference across a hundred million people multiple times a day for a year? It was always the second one, and doing the training cheaper makes it even more so.
But this implies that we could use those same resources to train even bigger models, right? Except that you then have the same problem. You have a bigger model, maybe it's better, but if you've made inference cost linearly more because of the size and the size is now 40x bigger, you now need that much more compute for inference.