Re:Show HN: Exafunction NLP – Run models like GPT-J for 6x c...
by varunkmohan on Tuesday September 28, 2021 @02:34PM
(#28685947)
Hi HN, I’m Varun, cofounder of Exafunction. We’re excited about all the use cases of large NLP models like text generation and language comprehension, but found the existing offerings to be quite expensive. Our goal is to serve these models in a cost effective way at scale.
The first model we’re starting with is GPT-J, the recently released open source large language model by EleutherAI. It’s comparable to OpenAI’s GPT-3 Curie in performance. With lots of optimizations, we are able to serve the model for 6x cheaper per token compared to OpenAI.
We’re also looking into supporting custom and fine-tuned models since it’s required to get good performance for a lot of applications. Especially for smaller models like BERT, we’re excited about the cost savings we can deliver for users who use many different models. In this case, we’re 100 - 1000x cheaper than others offering similar inference APIs like Hugging Face.
We hope this is interesting to you all and please email me at [email protected] if there’s a model you’d like to see supported.
Re:Show HN: Exafunction NLP – Run models like GPT-J for 6x c...
by d13 on Wednesday September 29, 2021 @03:12AM
(#28691069)
But how fast? I see other companies advertising 1.5 second response times for GPT-J, but a now assume that’s average per token, as for, say, a 200 word prompt response times can be well over a minute during heavy use periods like weekends when everyone is hitting their side projects.
Re:Show HN: Exafunction NLP – Run models like GPT-J for 6x c...
by varunkmohan on Wednesday September 29, 2021 @06:57PM
(#28701077)
For the public API, we should be under 100ms per token but don't have a strict guarantee. If you have a strict SLA, you can talk to us and we can get it to as low as 20ms per token at high load.
Here's a wonderfully written breakdown of some functionality that can be achieved with gpt-j and gpt-neo - it's not ExaFunction specific, but the language models are the same.
The exafunction api is fairly simple and powerful. Kudos!
Re:Show HN: Exafunction NLP – Run models like GPT-J for 6x c...
by varunkmohan on Tuesday September 28, 2021 @06:59PM
(#28688453)
Unlike OpenAI, which is serving a closed-source model, anyone can validate for themselves (even without using our service) whether open-source models like GPT-J are suitable for a particular application. We'd like to work with the open source community to make this kind of technology more accessible, essentially as an infrastructure provider.
If you have some specific use case in mind feel free to shoot us an email at [email protected]
Re:Show HN: Exafunction NLP – Run models like GPT-J for 6x c...
by ml_hardware on Tuesday September 28, 2021 @03:10PM
(#28686260)
Wow! How are you able to achieve the cost reductions? Is it different hardware, software optimizations, or both? Also does this suggest that OpenAI is charging 6x markups on their models... >:(
Re:Show HN: Exafunction NLP – Run models like GPT-J for 6x c...
by LuisMondragon on Tuesday September 28, 2021 @02:58PM
(#28686165)
Awesome. I am testing HuggingFace's API and evaluating if I should pay for a dev account. I will evaluate Exafunction also, it looks interesting. Specifically, I'm interested in using the multilingual LaBSE model. Do you support this model?
Re:Show HN: Exafunction NLP – Run models like GPT-J for 6x c...
by varunkmohan on Tuesday September 28, 2021 @03:02PM
(#28686188)
We currently do not support this model but are rolling out support for more models. Please add your request to the waitlist and we'll try to support your model as soon as possible!
Re:Show HN: Exafunction NLP – Run models like GPT-J for 6x c...
by suryabhupa on Tuesday September 28, 2021 @04:23PM
(#28687050)
This is really remarkable! How hard do you think it will be to support new models, i.e. does the tooling you’ve built generalize to you being able to serve other large scale models easily?
Re:Show HN: Exafunction NLP – Run models like GPT-J for 6x c...
by varunkmohan on Tuesday September 28, 2021 @03:32PM
(#28686502)
There's some secret sauce behind it, but mostly just using relatively inexpensive cloud inference hardware very effectively. It turns out most of the common NLP frameworks leave a good deal of performance on the table, not to mention the importance of minimizing cloud costs through general methods like using spot instances.
The first model we’re starting with is GPT-J, the recently released open source large language model by EleutherAI. It’s comparable to OpenAI’s GPT-3 Curie in performance. With lots of optimizations, we are able to serve the model for 6x cheaper per token compared to OpenAI.
We have a simple HTTPS API that you can try out for free. Here’s a link with the details - https://www.exafunction.com/nlp-api
We’re also looking into supporting custom and fine-tuned models since it’s required to get good performance for a lot of applications. Especially for smaller models like BERT, we’re excited about the cost savings we can deliver for users who use many different models. In this case, we’re 100 - 1000x cheaper than others offering similar inference APIs like Hugging Face.
We hope this is interesting to you all and please email me at [email protected] if there’s a model you’d like to see supported.
Please could you explain some use cases?
OpenAI is very strict with GPT-3. Say a company wanted to use Exafunction for email interactions or Discord interactions or even SMS interactions.
What would be the setup? How would that work with Exafunction?
Thank YOU!
Here's a wonderfully written breakdown of some functionality that can be achieved with gpt-j and gpt-neo - it's not ExaFunction specific, but the language models are the same.
The exafunction api is fairly simple and powerful. Kudos!
If you have some specific use case in mind feel free to shoot us an email at [email protected]