Re:Mixtral 8x22B
by afro88 on Wednesday April 17, 2024 @02:30PM
(#40068464)
How useful is such a large input window when most of the middle isn't really used? I'm thinking mostly about coding. But wheb putting even say 20k tokens into the input, a good chunk doesn't seem to be "remembered" or used for the output
[ Parent
| on HN ]
Re:Mixtral 8x22B
by jakderrida on Wednesday April 17, 2024 @06:07PM
(#40070568)
While you're 100% correct, they are working on ways to make the middle useful, such a "Needle in a Haystack" testing. When we say we wish for context length that large, I think it's implied we mean functionally. But you do make a really great point.