News for hackers. Stuff that matters.

A comment on Show HN: Open-source real-time talk-to-AI wearable device for few $. Parent comment

Re:Show HN: Open-source real-time talk-to-AI wearable device... by _joel on Saturday October 05, 2024 @03:09PM (#41752156)
You can make an OpenAI compatible local server using LMStudio[1] and load any model you want. It'd have to be on another host though, the s3 has some inference capabilities with addons afair, but nowhere enough grunt to run locally at any usable token/s

[1] https://lmstudio.ai/

[ Parent | on HN ]
Re:Show HN: Open-source real-time talk-to-AI wearable device... by homarp on Sunday October 06, 2024 @08:19AM (#41756623)
or with open source tools like

llama.cpp - https://github.com/ggerganov/llama.cpp/blob/master/examples/...

or

mistral.rs - https://github.com/EricLBuehler/mistral.rs/blob/master/docs/...

lmstudio and ollama use llama.cpp underneath. cut the middle man

[ Parent | on HN ]