Building LLMs from the Ground Up: A 3-Hour Coding Workshop

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by abusaidm on Saturday August 31, 2024 @06:34PM (#41412552)

Nice write up Sebastian, looking forward to the book. There are lots of details on the LLM and how it’s composed, would be great if you can expand on how Llama and OpenAI could be cleaning and structuring their training data given it seems this is where the battle is heading in the long run.

[ Parent | on HN ]

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by rakahn on Saturday August 31, 2024 @09:55PM (#41413577)

Yes. Would love to read that.

[ Parent | on HN ]

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by rahimnathwani on Saturday August 31, 2024 @10:59PM (#41413889)

  how Llama and OpenAI could be cleaning and structuring their training data

If you're interested in this, there are several sections in the Llama paper you will likely enjoy:

https://ai.meta.com/research/publications/the-llama-3-herd-o...

[ Parent | on HN ]

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by kbrkbr on Sunday September 01, 2024 @10:22AM (#41417188)

But isn't it the beauty of llm's that they need comparably little preparation (unstructured text as input) and pick the features on their own so to say?

edit: grammar

[ Parent | on HN ]

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by aDyslecticCrow on Monday September 02, 2024 @10:19AM (#41425582)

Yes, if you want an LLM that doesn't listen to instructions and just endlessly babbles about anything and everything.

What turned GPT into chatGPT was a lot of structured training with human feedback.

[ Parent | on HN ]

Re:Building LLMs from the Ground Up: A 3-Hour Coding Workshop by rahimnathwani on Tuesday September 03, 2024 @11:47PM (#41441714)

Exactly. Section 4.3.7 briefly explains how they trained the model to better follow instructions ('steerability').

[ Parent | on HN ]