News for hackers. Stuff that matters.
Re:AMD acquires Taalas to boost inference performance by etc... by bastawhiz on Thursday August 06, 2026 @06:39PM (#49203604)
The weights might fit in cache, if you're using a small model. If you wanted to have a 20B+ parameter model, that's just going in RAM. You could put more RAM in the device and pay the perf cost or have a dedicated chip. Most devices already have a dedicated chip, this just changes which silicon you're spending the money on.
Re:AMD acquires Taalas to boost inference performance by etc... by wmf on Thursday August 06, 2026 @08:07PM (#49204361)
That math doesn't really work.

8B model (FP4) = 4 GB DRAM = 32 Gb DRAM = 80 mm2

8B model (Taalas) = 4 GB ROM = ~800 mm2