News for hackers. Stuff that matters.
Re:RWKV: Reinventing RNNs for the Transformer Era by chaxor on Monday May 22, 2023 @11:35PM (#36039490)
I'll give a very simplistic answer to your question and skirt some detail. Essentially, Transformers can 'look at' everything within the context with perfect fidelity (only one 'hop' between tokens to 'get the info'), while RNNs struggle with this because the information gets 'muddied' with all of the tokens in between. Like telephone between you and one friend (transformer) vs you and a sea of people (RNN).

Hopefully this gets the point across without the mathematics and a precise description.

Re:RWKV: Reinventing RNNs for the Transformer Era by pico_creator on Tuesday May 23, 2023 @12:05AM (#36039683)
Yea using the analogy of the eyes can see the entire document

Vs

I need to memorise everything as it’s spoken out, and then answer on it

Is a good approximate on the difference.

The kicker though as people pointed out, there are individuals on earth who can memorise things to great length. So the same will apply here (need more training)

It’s also why it is recommended by the community to ask the question first, then put your context document. In a document QnA task

Re:RWKV: Reinventing RNNs for the Transformer Era by soulofmischief on Tuesday May 23, 2023 @01:55AM (#36040405)
In the shower today I was thinking about Jungian archetypes as a visual programming language for the subconscious.

For example, if someone says, "I should be a better mother", and hold this impression in their mind over time, the associations their brain makes with that image and various behaviors can be used as a guiding force during the physical reorganization which occurs when one creates a new habit, or adjusts a behavior or muscle memory.

And these associations depend upon observations made by the individual, which they have either been taught or innately know can be categorized as mother-like. The brain is typically trained to make this association in the case of the mother by mirroring and attaching to the first female who exhibits nursing behavior. These archetypes are a visual representation of deeper mental models which we are primed to carry on between generations.

And then I wondered how this might apply to neural models, if some system of archetypes naturally emerges, if they're useful for programming or memorization/compression, or how we might explore imbuing existing models with an archetypical system.