Re:Our next-generation model: Gemini 1.5
by danielmarkbruce on Thursday February 15, 2024 @05:06PM
(#39389775)
Manual testing on complex documents. A big legal contract for example. An issue can be referred to in 7 different places in a 100 page document. Does it give a coherent answer?
A handful of examples show whether it can do it. For example, GPT-4 turbo is downright awful at something like that.
Re:Our next-generation model: Gemini 1.5
by somenameforme on Friday February 16, 2024 @12:17AM
(#39393449)
You need to use relevant data. The question isn't random sorting/pruning, but being able to apply large numbers of related hints/references/definitions in a meaningful way. To me this would be the entire point of a large context window. For entirely different topics you can always just start a new instance.
What comes to my mind: run the usual gamut of tests, but with the excess context window saturated with irrelevant(?) data. Measure test answer accuracy/verbosity as a function of context saturation percentage. If there's little correlation between these two variables (e.g. 9% saturation is just as accurate/succinct as 99% saturation), then "muddiness" isn't an issue.