News for hackers. Stuff that matters.
Re:Ask HN: Is It Possible?, Gemini's Long Context Moe Archit... by deazy on Sunday May 18, 2025 @12:53PM (#44022661)
Basically this is what I think it is,

Shared context that can act as independent shards of (mini) contexts, i.e Sub-global attention blocks or "sub-context experts" that can operate somewhat independently and then scale up or compose into a larger global attention as a paradigm for handling extremely long contexts.

Trying to see if this can be tested in some way at small scale, its worth a try if it can work, but requires some engineering to make it possible.

There are no comments on this comment yet.