The tide is shifting: 1.3B outperforms 7B Llama 2

Re:The tide is shifting: 1.3B outperforms 7B Llama 2 by tayo42 on Tuesday September 12, 2023 @12:54PM (#37484156)

it might not exactly be a waste, when you are inputting your training data you need some context on what the source is and then add a weight to how valid it is. That is how we learn, if i talk to a phd on some topic, I would pay close attention to what they say and give it more significance, maybe even write it down, compared to a youtube video on that topic, compared to a guy on a street talking about that same topic, ill probably nod my head and walk away and try to forget as soon as i can. but there probably still is something valid to learn from that last conversation, how a certain type of person structures sentences, responds, the words used etc...

do these llms get trained with something like a credibility weight on the training data? that was it seems they did in this paper, just manually curated that

[ Parent | on HN ]

Re:The tide is shifting: 1.3B outperforms 7B Llama 2 by mpixel on Tuesday September 12, 2023 @01:15PM (#37484442)

Do people want that though? I don't want phd level responses for my queries. I want it to be better than what I could come up in a minute or by searching half an hour. Rather than some highly advanced highly detailed response I could probably not understand if the topic is not something I'm sufficient in to begin with.

Think common use cases. A lot of users are students, do I want it to write an essay like a linguist? Or solve my homework using the better but more advanced techniques and style?

[ Parent | on HN ]

Re:The tide is shifting: 1.3B outperforms 7B Llama 2 by tayo42 on Tuesday September 12, 2023 @01:32PM (#37484627)

I think they want it. What do you think the most perfect or ideal question answer-er or teacher is? It would probably be an expert in the field, but also with the ability to deliver that content in a level appropriate way for the recipient/student. Unreasonable for the most part, we get away with good enough in the real world. This is a skill we all try to learn though. Like when you need to give a technical presentation to non technical audience

If you have input data that includes a high rated reddit eli5 question, the content of that answer might be hard to verify, the style and way its delivered would be ideal to keep around in the training data. on the other side, technical in-depth answers have content that is worth keeping around, the style of its delivery would be very specific.

Keeping the entire internet around in your training data would still give you access to all these types of delivery still. hope that makes sense.

[ Parent | on HN ]

Re:The tide is shifting: 1.3B outperforms 7B Llama 2 by Valgrim on Tuesday September 12, 2023 @06:39PM (#37489565)

Maybe the answer is multi-step: first use curated primary sources, e.g. scientific papers. Then reinforce using well written summaries, perhaps by actual models or well graded student papers. Finally, somehow apply negative weights using wrong answers only. Bonus points if you can automate the whole process

[ Parent | on HN ]

Re:The tide is shifting: 1.3B outperforms 7B Llama 2 by Bjartr on Tuesday September 12, 2023 @03:01PM (#37486126)

It's a lot easier to add a pass to simplify an explanation by rephrasing or eliding information than it is to smarten up an overly simplified answer.

You do want the underlying model to be capable of the advanced answers, since if it is, it can be used to supply simple answers. You can't make that work the other way around in the same way.

[ Parent | on HN ]