News for hackers. Stuff that matters.
Re:Mixtral 8x22B by afro88 on Wednesday April 17, 2024 @02:30PM (#40068464)
How useful is such a large input window when most of the middle isn't really used? I'm thinking mostly about coding. But wheb putting even say 20k tokens into the input, a good chunk doesn't seem to be "remembered" or used for the output
Re:Mixtral 8x22B by jakderrida on Wednesday April 17, 2024 @06:07PM (#40070568)
While you're 100% correct, they are working on ways to make the middle useful, such a "Needle in a Haystack" testing. When we say we wish for context length that large, I think it's implied we mean functionally. But you do make a really great point.