1: Narrow Expertise: AIs usually target different things, so for example, in the context of LLMs, one might target being factually correct, so a knowledge benchmark is better suited, while another might target being good at coding, so a coding benchmark is better suited.

2: Answer Memorization: Once a benchmark has been live for a long time, AI's might start to learn from it, as it might be on various articles, and may have leaked into the training data, so unless a new benchmark is used, it would just be outputting memorised (overfitted) answers rather than being actually good at what the benchmark aims to measure.

3: Marketing: AI companies like to make their product look good (just like any other company), so they will often perform a ton of benchmarks, and only display the results where the AI performed best, to make it look like the AI is good in general.