If AI writes your code, why use Python?

Re:If AI writes your code, why use Python? by fulafel on Tuesday May 12, 2026 @01:03AM (#48104386)

What would comparing rates across languages tell in the context of this benchmark? Are the tasks the same or robustly difficulty-normalized across the languages?

Also somehow the 2 language comparison graphs (avg percentile and success rate) rank Python in dramatically different positions, with Python outranking Rust and Java in the success rate. What does the avg percentile mean in this context?

[ Parent | on HN ]

Re:If AI writes your code, why use Python? by gertlabs on Tuesday May 12, 2026 @03:21AM (#48105251)

Success rate measures the amount of code submissions that played the game/environment without failing (compilation, breaking game rules, violating sandbox, etc.), so it makes sense Python would do better there.

Percentile compares only the submissions that didn't hard-fail. So they are a bit different, and we incorporate them both into the combined score.

[ Parent | on HN ]

Re:If AI writes your code, why use Python? by Yokohiii on Tuesday May 12, 2026 @10:41AM (#48109009)

Comparing rust to javascript, the gscore is rather similar in distribution, while python falls off. I don't see why python should be so much worse?

[ Parent | on HN ]

Re:If AI writes your code, why use Python? by gertlabs on Tuesday May 12, 2026 @11:50AM (#48110028)

This was an unexpected result, and it held up under large sample sizes.

[ Parent | on HN ]

Re:If AI writes your code, why use Python? by Yokohiii on Wednesday May 13, 2026 @05:04AM (#48119497)

Did you try to analyze a few of the failures? I would be very intrigued here.

[ Parent | on HN ]