News for hackers. Stuff that matters.

A comment on Llama3 implemented from scratch. Parent comment

Re:Llama3 implemented from scratch by netdevnet on Monday May 20, 2024 @04:28AM (#40413456)
> What makes LLMs difficult is getting access to all the hardware to train them, getting the data, and being able to preprocess that data.

Yes, that's my opinion too. GAOs (Grassroots AI Organisations) are constrained by access to data and the hardware needed to process the data and train the model on it. I look forward to a future where GAOs will crowdsource their computations in the same way many science labs borrow computing power from people around the world.

[ Parent | on HN ]
Re:Llama3 implemented from scratch by miki123211 on Monday May 20, 2024 @09:50AM (#40415466)
This is hard because you need high bandwidth between the GPUs in your cluster, bandwidth far higher than broadband could provide. I'm not even sure whether the time spend synchronizing between far-away machines would offset the increase in computational power.
[ Parent | on HN ]