Re:Cerebras launches inference for Llama 3.1; benchmarked at...
by wkat4242 on Tuesday August 27, 2024 @06:57PM
(#41374173)
These wafers only have 44GB of RAM though. Very curious why the quantity is so low considering the chips are absolutely massive. It's SRAM though so very fast, comparable to cache in a modern CPU. But I assume being fast and loading the whole model there is the point.
Re:Cerebras launches inference for Llama 3.1; benchmarked at...
by adgjlsfhk1 on Tuesday August 27, 2024 @09:58PM
(#41375253)
probably largely because dram is a lot more dense than sea.
70B does significantly better in this regard. Nowhere close to perfection, but the frequency of WTFs about LLM's output are [subjectively] drastically lower.
Speed can be useful in RP if we'd run multiple LLM-based agents (like "plot", "goal checker", "inventory", "validation", "narrator") that function call each other to achieve some goal.