News for hackers. Stuff that matters.
Re:Meta Open-Sources Megalodon LLM for Efficient Long Sequen... by causalmodels on Tuesday June 11, 2024 @12:08PM (#40647996)
Re:Meta Open-Sources Megalodon LLM for Efficient Long Sequen... by skilled on Tuesday June 11, 2024 @10:50AM (#40646848)
Related:

Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length - https://news.ycombinator.com/item?id=40054901 - April 2024 (28 comments)

Re:Meta Open-Sources Megalodon LLM for Efficient Long Sequen... by koolala on Tuesday June 11, 2024 @12:53PM (#40648620)
yay dinosaurs!
Re:Meta Open-Sources Megalodon LLM for Efficient Long Sequen... by meindnoch on Tuesday June 11, 2024 @01:07PM (#40648817)
Megalodon was not a dinosaur. It was a fish (shark).
Re:Meta Open-Sources Megalodon LLM for Efficient Long Sequen... by lochiero on Tuesday June 11, 2024 @12:53PM (#40648621)
So old... Another epoch talking about Llama2
Re:Meta Open-Sources Megalodon LLM for Efficient Long Sequen... by arjvik on Tuesday June 11, 2024 @01:35PM (#40649162)
> For what it's worth, RWKV's website on that matter mentions that yes it's bad on recall, but for the vast majority of tasks you can just ask the question before the content, and it'll handle the task just fine.
Re:Meta Open-Sources Megalodon LLM for Efficient Long Sequen... by amluto on Tuesday June 11, 2024 @01:39PM (#40649218)
If you ask me a question before giving me the material being queried, I do better, too! It’s the difference between an open-book and closed-book test.

The Transformer read-then-query model is, IMO, a bit odd. It preprocesses the input and then people want it to be able to answer any question about the input and would also like the query to run in time independent of the input length, and people are sad that the best models take time linear in the input length. No kidding: that’s the algorithmic complexity of even a straightforward, non-ML approach!

Re:Meta Open-Sources Megalodon LLM for Efficient Long Sequen... by 1024core on Tuesday June 11, 2024 @02:50PM (#40649985)
MEGALODON builds on the research team's previous model, MEGA (exponential moving average with gated attention)

I like how the team gingerly avoided calling the previous model "MAGA" ... :-D

Re:Meta Open-Sources Megalodon LLM for Efficient Long Sequen... by codetrotter on Tuesday June 11, 2024 @02:57PM (#40650068)
I could give two sh*ts about Mr D.T. but I’m more curious to see the reaction of one Kim Dotcom, aka the Mega dude himself.
Re:Meta Open-Sources Megalodon LLM for Efficient Long Sequen... by ai_what on Tuesday June 11, 2024 @03:02PM (#40650130)
Didn't this happen in April?

Edit: Yes it did, https://github.com/XuezheMax/megalodon