Re:Show HN: GPU-Accelerated Inference Hosting
by spullara on Tuesday July 13, 2021 @12:36AM
(#27817669)
Does it need to reinitialize for each request or is there a warm start / cold start model like lambda? I don't really understand how you can charge per request.
[ Parent
| on HN ]
Re:Show HN: GPU-Accelerated Inference Hosting
by maldeh on Tuesday July 13, 2021 @01:18AM
(#27817810)
The pricing appears to be static per model with a ceiling on the monthly request count, not charged per request.
Edit: Actually, I didn't spot the free tier of 1000 requests. I wonder how you avoid the problem of a lot of users leaving defunct/disused models running while still keeping them hot - presumably some kind of limit to the model count?
Re:Show HN: GPU-Accelerated Inference Hosting
by theo31 on Tuesday July 13, 2021 @12:57AM
(#27817746)
There is no cold start! We keep your service hot all the time.
Re:Show HN: GPU-Accelerated Inference Hosting
by spullara on Tuesday July 13, 2021 @03:18AM
(#27818353)
Well, I guess I know where I am going to host GPT-J-6B then. I don't think it is sustainable.
Re:Show HN: GPU-Accelerated Inference Hosting
by marcooliv on Tuesday July 13, 2021 @08:37AM
(#27820154)
How are you planning to put a gpt whatever when the service clearly have a model size limit?!