The specification constrains the implementation details. Repeating the demonstrated example, again: filename must be lowercase is a perfect example of such a specification. You can have a hundred different tasks that require different things but all require that common specification.

> there is nothing deterministic about using llm's for codegen

And if you're okay with that, you're more like a farmer eating the slop meant for his pigs than an engineer. Using Sol Max, you can get near-deterministic output for a much wider range of tasks and specifications. Not all specifications; at a certain degree of sufficiently detailed specification it will begin failing again. But some, which is better than none.

i just asked luna to write a function that transformed a file path, and threw when the filename wasn't lowercase. completed that task just fine. i see no reason it wouldn't work if i asked it for 99 other functions with that same constraint.

you just aren't using it right man. stick to the big guns, you apparently need it.

You are sorely missing the point. You made the constraint the specific and only focus of the task. The example from the OP pertains to non-filename-related-tasks that have an additional specification added on to them. If your filenames need to be lowercase for all tasks, not just specifically filename-related tasks, you start running into problems.

> you're expecting to use it without steering

If I have to manually "steer" every mistake it makes on every trivial task, instead of being able to deterministically eliminate classes of failures so that fixing the task one time ensures that failure never happens again on similar future tasks, it is in fact faster to just type the fucking code myself.

op's tests are underspecified one-shot tasks. it's entirely expected that smaller models perform poorly.

but your claim was about file name invariants when generating code, especially small amounts of code, which is not what they were doing with DeepSWE, so you've confused two different things.

anyways, this is a pointless argument. if you can't figure out how to effectively use smaller models in your workflow, fine. but don't come on here talking shit about people who can. have fun slopping out yet another ai harness at your startup.

While OP's tests are indeed of the vague DeepSWE variety, I still reference them because the exact same failure mode occurs in my own tests even for trivial scope tasks, down to the level of having one single constraint for a variety of <10 LoC tasks. As you say, though, this is a pointless argument, and I'm not going to burn my existing benchmarks to try to convince anyone.

> it is in fact faster to just type the fucking code myself.

shhhhhhh, not so loud!