News for hackers. Stuff that matters.
Re:Gemini 1.5 Pro by ein0p on Thursday February 15, 2024 @01:04PM (#39386022)
Not "almost", but certainly. Dense attention is quadratic, not even Google would be able to run it at an acceptable speed. Their model is not recurrent - they did not have the time yet (or resources - believe it or not, Google of 2023-24 is very compute constrained) to train newer SSM or recurrent based models at practical parameter counts. Then there's the fact that those models are far harder to train due to instabilities, which is one of the reasons why you don't yet see FOSS recurrent/SSM models that are SOTA at their size or tokens/sec. With sparse attention, however, long context recall will be far from perfect, and the longer the context the worse the recall. That's better than no recall at all (as in a fully dense attention model which will simply lop off the preceding parts of the conversation), but not by a hell of a lot.
Re:Gemini 1.5 Pro by kiraaa on Thursday February 15, 2024 @06:40PM (#39390886)
maybe they are using ring attention, on top of their 128k model.
Re:Gemini 1.5 Pro by ein0p on Thursday February 15, 2024 @09:18PM (#39392313)
More likely some clever take on RAG. There’s no way that 1M context is all available at all times. More likely parts of it are retrievable on demand. Hence the retrieval-like use cases you see in the demos. The goal is to find a thing, not to find patterns at a distance
Re:Gemini 1.5 Pro by kiraaa on Friday February 16, 2024 @05:32AM (#39395295)
could be true, we can only speculate.