Re:Addressing AI’s energy cost
by Voloskaya on Wednesday June 08, 2022 @02:42PM
(#31672134)
CPU inference of even the smaller version of GPT-3 would be way too slow for a public API.
What CPU did you benchmark on that gave you a cheaper inference price than GPU? In my experience, for GPT like transformers, they don’t come anywhere near what you can squeeze out of something like a Nvidia T4 in terms of either performance or $/token.
What CPU did you benchmark on that gave you a cheaper inference price than GPU? In my experience, for GPT like transformers, they don’t come anywhere near what you can squeeze out of something like a Nvidia T4 in terms of either performance or $/token.