- Communication speed is one of the biggest bottlenecks to large models, so their bandwidth of 4TBps is very smart.
- They claim a 1.3x perf/watt improvement, which is not really that great for ASICs compared to GPUs. Perf/watt is probably the most important number in datacenters.
- They only use SRAM, no DRAM. This is a huge mistake, which limits their model size. You can only fit a ~10GB model inside a single tile, versus 80GB models for a single A100 GPU.
- Software / compiler stack is as or more important than the hardware itself, because it dictates how much real performance you can squeeze out of the chips. I think Tesla will need to heavily focus on this area before getting anywhere close to real-world GPU performance.
This seems pretty interesting. They must have had a reason for this approach. Maybe it doesn't matter as much for their usecase.
They clearly had the option of adding DRAM if they wanted to. Why not do it?
In Q+A they said the Dojo stuff was an aspirational vision, not a production-ready design
Elon is a self parody at this point.
And Dojo is clearly advanced state, as they show the hardware and showed how far they are. They haven't built the full rack of course.
> Elon is a self parody at this point.
I think people who act like everything he says is some made up nonsense are a parody of themselves if anything.
There's a much higher chance there's a consumer fraud case on FSD than ten of these robots are ever sold
How did you get that interpretation from Elon saying they're aiming to have a prototype next year?
It's a recruiting event. Presumably in part to recruit people to work on this new project.
I have never really seen a system integrated like that? Are there other examples of people doing this? They seem to have designed some kind of sled that connect to power and liquid cooling system.
Can somebody with some expertise in large scale training compare this to state of the art?