Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generations based on LLaMa

Re:Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generat... by barking_biscuit on Tuesday March 28, 2023 @09:19PM (#35350747)

>Buying a "high end GPU" isn't buying a 4090 or even two, it's 250k on a DGX unit and putting it in a datacentre. You will probably be able to find a service that would sign a confidentiality agreement and provide you with this service for less than 250k.

I haven't started to play with LLMs locally in anger yet, but I was under the impression that you could use a 4090 in combination with FlexGen to achieve this rather than having to buy special hardware?

[ Parent | on HN ]

Re:Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generat... by tetete on Tuesday March 28, 2023 @10:38PM (#35351564)

With 8 bit training you can do ~13B pram LLM on 3090/4090. https://huggingface.co/blog/trl-peft

But it is pretty cheap to rent something at vast.ai or whatever to get 40GB for a final run.

[ Parent | on HN ]

Re:Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generat... by nullsense on Wednesday March 29, 2023 @06:12AM (#35354998)

Awesome. Between crypto hype in 2017 and AI hype in 2023 I've acquired a collection of 2x 1080ti, an RTX 3060 and an RTX 4090. All together it's a total of 58GB of VRAM. Is there a way I can pool it all across a distributed cluster of 2 machines for doing anything? I'm assuming it would bottleneck on both network speeds and the slowest GPUs if it's possible at all...

[ Parent | on HN ]

Re:Gpt4all: A chatbot trained on ~800k GPT-3.5-Turbo Generat... by all2 on Thursday March 30, 2023 @08:13PM (#35380633)

Could you package all that compute into a "virtual" graphics card?

[ Parent | on HN ]