News for hackers. Stuff that matters.

A comment on Sohu: The First Transformer ASIC.

Re:Sohu: The First Transformer ASIC by mysterEFrank on Tuesday June 25, 2024 @06:56PM (#40794645)
Do they bake in the actual weights or the architecture? If it's just the architecture I don't understand where a speedup that considerable can come from.
[ Parent | on HN ]
Re:Sohu: The First Transformer ASIC by htthbjk on Tuesday June 25, 2024 @07:06PM (#40794720)
They mention something about not wasting space for unnecessary memory.
[ Parent | on HN ]
Re:Sohu: The First Transformer ASIC by jasonni on Wednesday June 26, 2024 @04:04AM (#40797481)
from their announcement, "Isn’t inference bottlenecked on memory bandwidth, not compute?", it seems weights are still in memory. It may have limit onchip cache for computing. Input tokens go through a batch pipeline to relieve memory bottleneck. Similar to Groq.
[ Parent | on HN ]