Sounds like they don’t even need to pay for AI. Maybe a deterministic script that does some basic quality checks like verifying that the linked repository isn’t day old slop, and auto-rejects the ones that don’t pass muster would be better. For starters, then you could have well-established and documented rules. Which people generally seem to prefer over AI referees that reject things for opaque and confusing reasons (e.g., YouTube moderation) and can’t be reasoned with.

I appreciate the suggestion but in this case I don't believe the solution to AI problem is using more AI. I personally believe what makes these lists "awesome" is the human element, not some deterministic checks that another agent could run.

IMO it's just like using an anti-spam filter in your email. An AI agent should never decide "Yes, this is awesome" but it surely can decide "No, this is obvious trash". Sometimes the simplest rules are best - "Did they follow the PR template? No? Ok, auto-close with a message saying that AI submissions are not allowed"

You don't need an agent to check if the PR matches the template.

The point is: it’s easier on the humans if submissions not following the template are auto-rejected.

Or do you mean just grep for “TEMPLATE-1.3” and if it’s not there, reject?

Yes - a deterministic tool is just fine for verifying that a template has been followed.

It's not quite as simple as "check for one line," but it's not hard to have a useful check that costs far fewer resources to run than an agent would.

The deterministic check could help you filter out the chaff - prevent the human element from getting DOSed.

That assumes one has the money to maintain an AI service/subscription, resources to run the prompt from, and the time to fine-tune the AI to be dependable with a high rate of accuracy in your criteria.

Maybe someone who has a lot of free time and high enthusiasm for AI might consider this to be worth chasing after, but anyone who has dealt with these whack-a-mole scenarios is going to admit the easiest thing to do is just not maintain the resource anymore. There are less-draining projects and endeavors to turn towards instead.

No. Judgment is the thing AI lacks the most. Sure, you can try to give it guidelines, but it sucks at interpreting them because it has no taste and no common sense. It is trivially influenced by bullshit submissions because it overindexes on text it sees and has no theory of the mind or sense of intent.

It is like a naive and prodigious intern with incredible intelligence and factual knowledge, but no sense of how the world works or what will actually fly with human stakeholders.