News for hackers. Stuff that matters.
Re:Show HN: LlamaGPT – Self-hosted, offline, private AI chat... by mayankchhabra on Wednesday August 16, 2023 @01:15PM (#37150241)
Yes! It can run on any home server or cloud server.
Re:Show HN: LlamaGPT – Self-hosted, offline, private AI chat... by ryanSrich on Wednesday August 16, 2023 @04:28PM (#37153319)
Interesting. I might try to get this to work on my NAS.
Re:Show HN: LlamaGPT – Self-hosted, offline, private AI chat... by reneberlin on Wednesday August 16, 2023 @06:42PM (#37154861)
Good luck! The token/sec will be under your expectations or it will overheat. You really shouldn't play games with your data-storage. You could try it with an old laptop to see how bad it performs. Ruining your NAS for this is a bit over the top to show, that "it worked somehow". But i don't know, maybe your NAS has a powerful processor and is tuned to the max and you have redundancy and don't care to loose a NAS? Or this was just a joke and i fell for it! ;)
Re:Show HN: LlamaGPT – Self-hosted, offline, private AI chat... by mayankchhabra on Wednesday August 16, 2023 @06:47PM (#37154903)
Not sure how powerful their NAS is, but on Umbrel Home (which has an N5105 CPU), it's pretty useable with ~3 tokens generated per second.