News for hackers. Stuff that matters.
Re:Fig Has Joined AWS by xcdzvyn on Monday August 28, 2023 @03:03PM (#37298925)
Just curious, do you have any intent on adding local model support?
Re:Fig Has Joined AWS by pbbakkum on Monday August 28, 2023 @03:11PM (#37299045)
I've experimented with it, the reason I haven't yet added it is that I want deployment to be seamless, and it's not trivial to ship a binary that would (without extra fuss or configuration) efficiently support Metal and CUDA, plus download the models in a graceful way. This is of course possible, but still hard, and not clear if it's the right place to spend energy. I'm curious how you think about it - is your primary desire to work offline or avoid sending data to OpenAI? Or both?
Re:Fig Has Joined AWS by xcdzvyn on Monday August 28, 2023 @11:18PM (#37303169)
The latter mostly. It's also free, uncensored, and can never disappear from under me.

FWIW, from my understanding llama.cpp is pretty easy to integrate and is reasonably fast for being API agnostic. Ollama embeds it, for example. No pressure, just pointing it out :)