News for hackers. Stuff that matters.

A comment on Show HN: Exafunction NLP – Run models like GPT-J for 6x cheaper than OpenAI. Parent comment

Re:Show HN: Exafunction NLP – Run models like GPT-J for 6x c... by d13 on Wednesday September 29, 2021 @03:12AM (#28691069)
But how fast? I see other companies advertising 1.5 second response times for GPT-J, but a now assume that’s average per token, as for, say, a 200 word prompt response times can be well over a minute during heavy use periods like weekends when everyone is hitting their side projects.
[ Parent | on HN ]
Re:Show HN: Exafunction NLP – Run models like GPT-J for 6x c... by varunkmohan on Wednesday September 29, 2021 @06:57PM (#28701077)
For the public API, we should be under 100ms per token but don't have a strict guarantee. If you have a strict SLA, you can talk to us and we can get it to as low as 20ms per token at high load.
[ Parent | on HN ]