Dalai: Automatically install, run, and play with LLaMA on your computer

Re:Dalai: Automatically install, run, and play with LLaMA on... by iambateman on Monday March 13, 2023 @12:20AM (#35131012)

If I was running in a server context, would the 50gb of ram be required to respond to one request, or can it be used to respond to multiple requests simultaneously?

[ Parent | on HN ]

Re:Dalai: Automatically install, run, and play with LLaMA on... by boredemployee on Monday March 13, 2023 @06:23AM (#35133652)

I'd assume that all the calculations used for 1 request would already eat up that amount of memory, but I could be wrong!

[ Parent | on HN ]

Re:Dalai: Automatically install, run, and play with LLaMA on... by lolinder on Friday March 17, 2023 @12:29AM (#35193132)

I'm very late to this question, but I believe that that amount is only required once, but the context tensor will need to be created per request. I haven't confirmed that, though.

[ Parent | on HN ]