News for hackers. Stuff that matters.

A comment on 70B Llama 2 at 35tokens/second on 4090. Parent comment

Re:70B Llama 2 at 35tokens/second on 4090 by quickthrower2 on Wednesday September 13, 2023 @05:45AM (#37494492)
Interested in this. Are you thinking of it being a SaaS type thing? I have some rudimentary experience, i.e. training NanoGPT on both Modal.com and Vast.ai (which is vastly cheaper, but privacy/cost trade off there).

It could be a layer on top of $gpu_cloud (so end user signs up for the cloud) and the service makes use of those resources.

[ Parent | on HN ]
Re:70B Llama 2 at 35tokens/second on 4090 by senseiV on Wednesday September 13, 2023 @02:20PM (#37499835)
Do you inference NanoGPT with say a Flask app? I made a WebUI for tiny Llama, but it doesn't work for NanoChatGPt
[ Parent | on HN ]
Re:70B Llama 2 at 35tokens/second on 4090 by quickthrower2 on Wednesday September 13, 2023 @07:07PM (#37502859)
Ha ha not sure yet, I have only done it from the command line. Still got a lot to learn. Building something like this will be good for that!
[ Parent | on HN ]