News for hackers. Stuff that matters.

A comment on LLMs use a surprisingly simple mechanism to retrieve some stored knowledge. Parent comment

Re:LLMs use a surprisingly simple mechanism to retrieve some... by refulgentis on Thursday March 28, 2024 @06:55PM (#39858416)
It doesn't seem promising, a one man band has been doing a quixotic quest based on intuition and it's gotten ~nowhere, and it's not for lack of interest in alternatives. There's never been a better time to have a different approach - is your metric "times I've seen it on HN with a convincing argument for it being promising?" -- I'm not embarrassed to admit that is/was mine, but alternatively, you're aware of recent breakthroughs I haven't seen.
[ Parent | on HN ]
Re:LLMs use a surprisingly simple mechanism to retrieve some... by dartos on Saturday March 30, 2024 @09:05AM (#39874493)
RWKV has shown that you can scale RNNs to large parameter counts.

The fact that one person (initially) was able to do it highlights how much low hanging fruit there is for non transformers.

Also, the fact that a small number of people designed, trained, and published 5 versions of a perfectly serviceable (as in has decent summarizing ability. The biggest LLM use case) model which doesn’t have the time complexity of transformers is a big deal.

[ Parent | on HN ]