Re:Show HN: Codeium – a free, ultrafast Copilot alternative
by mhuffman on Tuesday December 06, 2022 @05:35PM
(#33888089)
I have a ML infra that does between 1 million and 2 million inferences per day (granted it is not a chat bot or code generator, but it is legit ML inference in a similar mode) and it only cost $120 per month for the server (also runs multiple containers and is a web server), which mostly sits idle.
Re:Show HN: Codeium – a free, ultrafast Copilot alternative
by swyx on Wednesday December 07, 2022 @03:12PM
(#33899604)
right, but what size of model? i assume these large language models are... well, large
I'm really really curious as to the technical details of getting the inferences cheap and fast enough to enable a copilot like experience when you're not GitHub with inifinity resources.
Sam Altman has said that each chat in chatgpt is single digit cents per chat. GPT3 davinci is 0.02 per 1k tokens, not to mention the high latency. it'd be impossible to build "copilot for law" based on gpt3 because the UX wouldnt work with that latency or cost (you'd be running 100s of inferences if you count every debounced keystroke).
What cost per user/per inference/per day (however you think about it) do you need to get this down to? and what order of magnitude speedups did you need to accomplish to get this done?
(congrats btw, happy user here thanks to Prem)