gronky_

Mobile rendering of the post has some issues. Tables are overflowing and not responsive for example

Don’t know about other claws, with NanoClaw the agent can only rewrite code that runs inside the container.

You can see here that it’s only given write access to specific directories: https://github.com/qwibitai/nanoclaw/blob/8f91d3be576b830081...

True. There’s a setting for Claude code though where you can add apiKeyHelper which is a script you add that gets the token for Claude Code. I imagine you can use that but haven’t quite figured out how to wire it up

Keep in mind that this isn’t about users - the top agents on the leaderboard aren’t running an actual product on the benchmark.

If they are running their production product as is, then of course whatever is built into the product is fine.

This ok from your perspective then?

def make_pass@1_agent(agent, n):

    def retry_agent(problem):

        for attempt in range(n):

            result = agent(problem)

            if result.success:

                return result

        return result

    return retry_agent

It’s a pass@1 benchmark. When submitting you need to check a box that there was only 1 attempt per problem. See here for example: https://github.com/SWE-bench/experiments/pull/219

Building multiple attempts into your agent is stretching the rules, even if technically it’s acceptable

I’ve been running a bunch of coding agents on benchmarks recently as part of consulting, and this is actually much more impressive than it seems at first glance.

71.2% puts it at 5th, which is 4 points below the leader (four points is a lot) and just over 1% lower than Anthropic’s own submission for Claude Sonnet 4 - the same model these guys are running.

But the top rated submissions aren’t running production products. They generally have extensive scaffolding or harnesses that were built *specifically for SWE bench*, which kind of defeats the whole purpose of the benchmark.

Take for example Refact which is at #2 with 74.4%, they built a 2k lines of code framework around their agent specifically for SWE bench (https://github.com/smallcloudai/refact-bench/). It’s pretty elaborate, orchestrating multiple agents, with a debug agent that kicks in if the main agent fails. The debug agent analyzes the failure and gives insights to the main agent which tries again, so it’s effectively multiple attempts per problem.

If the results can be reproduced “out-of-the-box” with their coding agent like they claim, it puts it up there as one of the top 2-3 CLI agents available right now.

I see it a bit differently - LLMs are an incredible innovation but it’s hard to do anything useful with them without the right wrapper.

A good wrapper has deep domain knowledge baked into it, combined with automation and expert use of the LLM.

It maybe isn’t super innovative but it’s a bit of an art form and unlocks the utility of the underlying LLM

It will catch those sneaky bugs

Stating that Israel doesn’t have a right to exist has been recognized to be an antisemitic statement by many prominent institutions.

It’s a radical statement that effectively denies the rights of millions of people to exist and is especially problematic given the historical context of the establishment of Israel.

The statement gets thrown around so much in certain circles that it’s gotten normalized. You’ve apparently lost sight of or never stopped to think what actually means, to the point where you’re providing it as an example of an innocent statement that got you banned for no reason. Taking this statement out of radical activist circles and into the real world won’t go well.

Take some time to educate yourself and reflect on what it actually means.

Yes, it uses Supabase, doesn’t roll its own

I just tried the demo on the homepage and I don’t know what kind of sorcery this is but it’s blowing my mind.

I input a bunch of completely made up words (Quastral Syncing, Zarnix Meshing, HIBAX, Bilxer) and used them in a sentence and the model zero-shotted perfect speech recognition!

It’s so counterintuitive for me that this would work. I would have bet that you have to provide at least one audio sample in order for the model to recognize a word it was never trained on.

Providing it to the model in text modality and it being able to recognize it in the audio modality must be an emergent property.

They’re all in. They announced they’ll add support for it in the desktop app and the API in the coming months: https://x.com/OpenAIDevs/status/1904957755829481737

Currently supported in the Agents SDK https://openai.github.io/openai-agents-python/mcp/

we started using it recently at my work. the code changes walkthrough is nice