quelladora
LongProc looks like a fantastic test for a different but related problem, getting models to generate long answers. It seems to measure a skill the others don't. Meanwhile, RULER feels even more artificial than MRCR, since it's almost entirely focused on that simple "find the fact" skill.
But I think you're spot-on with the main takeaway, and the best frontier models are still struggling with long context. The DeepMind team points this out in the paper with that Pokemon example and the MRCR evaluation scores themselves.
See contextarena.ai and the original paper https://arxiv.org/abs/2409.12640
It also seems to match up well with evals like https://fiction.live/stories/Fiction-liveBench-Feb-21-2025/o...
The other evals you mention are not necessarily harder than this relatively simple one..