tadamcz

You're in luck! https://news.ycombinator.com/item?id=49254706

We studied this question pretty systematically in the MirrorCode paper [1], comparing Python, C, Rust, Go, OCaml, and Ada across 19 very long-horizon tasks, for Claude Opus 4.7 and GPT-5.5.

> In our results, there was little sign of inter-language differences in solve rates, for any model (Figure 5b). This suggests that AI models have learned generalized programming skills, rather than pattern-matching syntax. This does not mean that implementation language is irrelevant. Conditional on solving a target, we found a small effect on token usage: successful Python solutions tended to use fewer tokens than average, while successful Ada solutions tended to use more (Appendix C). We consider these to be small differences, given that these six programming languages vary widely in how concise they are, and in how much functionality is provided by their standard library (recall that agents cannot download dependencies in MirrorCode, they must solve the task using only the standard library).

In Appendix C, Ada tended to use only about 25% more tokens than the average language. Ada is a language used mainly in safety-critical aerospace and defense systems, which has ~200x less pre-training data available than C or Python.

We're also comparing more recent language models (on just Go vs Ada, for cost reasons), on our leaderboard [2].

[1] https://arxiv.org/pdf/2606.30182

[2] https://epoch.ai/MirrorCode#leaderboard

Hi HN! I'm the creator of MirrorCode, a benchmark of long-horizon SWE tasks.

In a MirrorCode task, AI models are tasked with reimplementing an entire program end-to-end, without access to the original source code. AI-generated solutions must match the original program’s stdout and stderr exactly on end-to-end tests. MirrorCode’s 25 target programs span different areas of computing: Unix utilities, data serialization and query tools, bioinformatics, interpreters, static analysis, cryptography, and compression.

IMO, the key tension in eval design is now: how to make tasks that are difficult for AI, but actually fair? My default expectation in 2026 when I see a coding eval with low scores is that the tasks turn out to be impossible.

In MirrorCode, we leverage reimplementation of existing software as the raw material to make tasks hard. But we also manually select each of the 25 target programs, and carefully collect the end-to-end tests, so that the task is actually achievable (if extremely difficult).

It's very easy for MirrorCode-style tasks to degenerate into absurd reverse-engineering: stuff you'd never do as a human engineer.

e.g. one of our target programs is the brotli compression library. But we explicitly test the decompression path only (and call the task "brotlid")

RFC 7932 only defines decompression: given compressed bytes, there's one correct output. In the other direction, there's no unique mapping.

So the output of a compressor is governed by thousands of internal efficiency heuristics (e.g. how far back to look for repeated strings). Replicating those perfectly from a black box is extreme and pointless reverse-engineering.

An aspect of MirrorCode I’m proud of: it’s the first benchmark I know of to implement a serious “security mindset”. The field fatalistically treats AI cheating as an unwinnable game of whack-a-mole. I think this is completely wrong.

My views: (1) in 2026, cheat-proofing must be a first-class consideration in benchmark design, not an afterthought (2) most benchmarks CAN be made secure, by using known primitives like containers

The problem is that benchmark authors aren’t even trying to be secure against human-level attackers, which AIs are.

Another key differentiator of MirrorCode: AIs are clearly told the scope of the task, rather than having to guess what they'll secretly be tested on.

We clarify task scope by showing AIs a list of test inputs (while also keeping some hidden to prevent AIs from cheating with a lookup table).

If this sounds too easy or like giving them the answer, I assure you, you're thinking about it wrong. e.g. perfectly matching gotree's behaviour does not become easy because I show you 1,899 inputs (!) that your program will be tested on.

Meanwhile, if you don't show test cases and just say "reimplement all of gotree" the task is dominated by an impossible guessing-game: what's actually in scope?

The Nexus format was only loosely specified in its original publication (which we give to AIs). Among other omissions, documentation does not mention that Nexus files may contain comments. Nexus comments are between brackets [], so they may appear in the middle of a line of data, unlike code comments. gotree _generally_ handles them without complaint, but errors with comments in certain locations.

I personally think guessing that comments exist, and all the ways comments must be handled, despite comments not even being mentioned in documentation, is functionally impossible. So the benchmark becomes dominated by the guessing-game, rather than actual implementation.

Human engineers gradually learn the scope of inputs a program should support thanks to external feedback from users of the program, they don't just think them all up in a vacuum before publication. Yet that's what some benchmarks test.

We’re releasing MirrorCode as open-source.

I agree it's a potentially big problem, affecting almost any benchmark out there. We discuss it briefly in "Appendix A: Contamination and memorization" https://epoch.ai/blog/mirrorcode-preliminary-results#appendi....

Ideally one would do these benchmarks with held-out proprietary software, but that comes with many practical concerns.

I think we're talking past each other here...

I should say one big difference is ProgramBench has 200 target programs while MirrorCode has about 30. We did many manual things to ensure task quality, that would have required huge resources to do at ProgramBench scale.

There's a more detailed description in "Appendix B: Qualitative discussion of the gotree task"

https://epoch.ai/blog/mirrorcode-preliminary-results#appendi...

> Eg cal is totally routine. I would expect most sophomores to be able to write a perfectly good cal.

This is incidental to the main disagreement, but btw I also doubt this.

Let's try and make the claim more precise. e.g. are you saying the average university undergraduate studying CS would reimplement cal from scratch (only stdlib), matching the output perfectly for all 1365 MirrorCode test cases, in (say) 3 days of full-time work (without AI assistance obviously)? I'd bet against it!

Here is the manual for the cal that we use: https://media.githubusercontent.com/media/epoch-research/Mir...

You can also look at a full transcript of an LLM solving the task: https://epochai-public-eval-logs-manual.s3.amazonaws.com/eva...

I don't think so. ProgramBench authors say no LLMs fully resolve any task, i.e. even the easiest tasks in their benchmark are unsolved. Whereas we found Opus 4.6 successfully reimplements almost every program up to gotree’s size (around 15-20 of them).

For Pkl, the preliminary results only went up to 1bn total tokens (costing $550, which would be cheap if LLMs could do the task). It might very well be solved at higher token budgets; see the report for more discussion of this.

The preliminary results are just on 4 targets. We have several Pkl-level and harder tasks in the full set which we're releasing soon.

In the following quote multiple things are not quite right:

> mostly involving higher-level languages, whereas ProgramBench are all very complex C programs (and much older programs with much more comprehensive test cases).

First, I think you're confusing the top-end of ProgramBench difficulty with the average. The quote in the OP is pretty clear that FFmpeg, SQLite, and PHP are the 3 hardest out of 200 in the benchmark:

> Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter.

Second, I don't see the relevance of C vs higher-level languages, how does this make ProgramBench harder?

Third, for the test cases, I think you might be labouring under a misapprehension about how MirrorCode works? MirrorCode uses end-to-end tests from a variety of sources (the original program’s test suites, real-world data, and LLM-assisted generation). End-to-end means the stdout/stderr has to match exactly for each test case.

Nice work once again from Ofir Press and team; this seems to be an idea that's in the air.

> Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task

Fwiw, this is very different from what we find in MirrorCode:

> Opus 4.6 successfully reimplements almost every program up to gotree’s size in our benchmark.

https://epoch.ai/blog/mirrorcode-preliminary-results

I don't have time right now to dig in to what could explain the difference (I'm working hard on getting the full MirrorCode out as soon as possible). But I suspect that the ProgramBench authors are either under-eliciting the AIs, or their tasks are unfair/impossible given the constraints, or both.

I hope to look more into it after releasing MirrorCode, and write up my conclusions.

Cool!

I couldn't reproduce your results, btw.

4 attempts, it answers "Sarah Constantin" in all 4. Is it possible your setup somehow leaks your name (or other identifying information) into the context?

I tried it on my writing, and it failed every time (I'm extremely obscure but have had a blog for 10 years). My verdict is that it guesses almost entirely based on the content/topic, not style.

https://bayes.net/prioritising-ai: Ben Garfinkel

https://bayes.net/normative-ethics: Richard Yetter Chappell

https://bayes.net/espai: David Owen, Ege Erdil

https://bayes.net/swebench-hack: Sayash Kapoor

https://bayes.net/frivolity: Amanda Askell

https://bayes.net/ps/: Pablo Stafforini

https://bayes.net/fertility-mortality/: Dynomight (the pseudonymous Substack/blog author)

Prompt was:

    Who likely wrote this? Don't search the web or databases. If you're not sure, just give me your best guess.

Using a security key instead of TOTP would have prevented this.

Using a security key as 2FA instead of TOTP would have prevented this attack, right?

If you maintain popular open source packages for the love of God get yourself a couple of security keys.

In July, I predicted future AI models would someday learn to cheat on SWE-bench by accessing future git history. Turns out, they were already doing it!

This doesn't explain how to get API access

The launch post for Gemma 3 says:

> use Gemma 3 with the Google GenAI SDK

https://blog.google/technology/developers/gemma-3/

Does this mean (serverless) API access? I haven't been able to do so or find docs that explain how to.

Other discussion: https://news.ycombinator.com/item?id=42972581

Hi, I'm the maintainer of the Epoch AI Benchmarking Hub.

We're building a transparent public dataset of AI model performance.

We log and publish every prompt and response -- not just aggregate scores. We even store and display the json bodies for every API call. Example: https://logs.epoch.ai/inspect-viewer/484131e0/viewer.html?lo...

Each evaluation is linked to detailed information we collect about the model, including its release date, the organization behind it, and in some cases our estimate of the amount of compute used to train the model.

At the moment, the database features results from two benchmarks:

- GPQA Diamond: This is a higher-quality, challenging subset of the GPQA benchmark, which tests models’ ability to answer PhD-level multiple choice questions about chemistry, physics, and biology.

- MATH Level 5: This is a subset of the hardest questions from the MATH benchmark, a dataset of high-school level competition math problems.

We plan to rapidly expand our suite of benchmarks to create a thorough picture of AI progress, by adding benchmarks such as FrontierMath, SWE-Bench-Verified, and SciCodeBench.

Announcement post: https://epoch.ai/blog/benchmarking-hub-update

Generally, we'll use the API provider's defaults.

For models we run ourselves from the weights, at the moment we'd use vLLM's defaults, but this may warrant more thought and adjustment. Other things being equal, I prefer to use an AI lab's API, with settings as vanilla as possible, so that we essentially defer to them on these judgments. For example, this is why we ran this Mistral model from Mistral's API instead of from the weights.

I believe the `temperature` parameter, for example, has different implementations across architectures/models, so it's not as simple as picking a single temperature number for all models.

However, I'm curious if you have further thoughts on how we should approach this.

By the way, in the log viewer UI, for any model call, you can click on the "API" button to see the payloads that were sent. In this case, you can see that we do not send any values to Mistral for `top_p`, `temperature`, etc.

Hi! I'm Tom, a machine learning engineer at the nonprofit research institute Epoch AI [0]. I've been working on building infrastructure to:

* run LLM evaluations systematically and at scale

* share the data with the public in a rigorous and transparent way

We use the UK government's Inspect [1] library to run the evaluations.

As soon as I saw this news on HN, I evaluated Mistral Small 3 on MATH [2] level 5 (hardest subset, 1,324 questions). I get an accuracy of 0.45 (± 0.011). We sample the LLM 8 times for each question, which lets us obtain less noisy estimates of mean accuracy, and measure the consistency of the LLM's answers. The 1,324*8=10,584 samples represent 8.5M tokens (2M in, 6.5M out).

You can see the full transcripts here in Inspect’s interactive interface: https://epoch.ai/inspect-viewer/484131e0/viewer?log_file=htt...

Note that MATH is a different benchmark from the MathInstruct [3] mentioned in the OP.

It's still early days for Epoch AI's benchmarking work. I'm developing a systematic database of evaluations run directly by us (so we can share the full details transparently), which we hope to release very soon.

[0]: https://epoch.ai/

[1]: https://github.com/UKGovernmentBEIS/inspect_ai

[2]: https://arxiv.org/abs/2103.03874

[3]: https://huggingface.co/datasets/TIGER-Lab/MathInstruct