Having it with clear hw requirements tiers is a nice differentiator. The only issue is that the benchmarks would 100% need to be closed, no other way around it. And then you have the issue of creating and curating good evals for every "stage" of the project. That's a hard task even for "honest" lab-internal evals. And you'd have to publish those evals after each round (for trust purposes), and start over for the next round. Doable, but it would cost a lot (probably more than the prize pools) and you'd have to keep doing this.
Yes, I hope the open model communities will someday be able to run current frontier models which will be able to handle autonomous tasks and the hardware to run it will be served at consumer level; however, like how recycling isn't profitable, no companies will fully commit to the movement. Don't get it twisted, I don't have a solution but maybe a global scale movement to liberate knowledge-library could suffice.
Every 6-12 months, give out $200K to the first model to hit a min threshold on a set of ~5-10 hard benchmarks (+ perhaps one secret benchmark) using a total of 16GB / 32GB / 64GB / 128GB of VRAM (at a min context length of 200K), then move the threshold up. Quantization etc. is dealers choice, it just needs to nail the benchmark on a reference machine by using exactly that much VRAM (no mapping to RAM / disk etc.)
You could crowdsource the funding, and cross subsidize by adding targeted prizes focused on corporate needs (the classic one is PDF processing benchmarks), and say that 25% of each corporate prize funding also flows into the general prize pool.
For a lot of these open-source model companies, it's less about the $s (though $200K is nothing to sneeze at), it's the clear recognition that helps their model efforts stand out, gain usage etc.