Ask HN: If you're running an LLM locally, what are you using?

Re:Ask HN: If you're running an LLM locally, what are you us... by smoldesu on Wednesday February 07, 2024 @01:18PM (#39292101)

I recently found out about llama-cpp's official "Server" function: https://github.com/ggerganov/llama.cpp/blob/master/examples/...

Works like a charm for my (simple) use case. I'm running it on an Always Free Oracle Ampere A1 instance with 4 cores and ~20gb of memory. (Obligatory "fuck Larry Ellison" here)

[ Parent | on HN ]

There are no comments on this comment yet.