pcwelder

Example could be, we'll update X criteria when top-N models are within 2 points of each other.

Since then none of the Google models have competed well with the others at their price points. Long context performance drop, instruction following issues, unnecessarily complex and sloppish outputs.

Argon is no different, it feels just an iteration better than 3.8 flash. I don't trust the AA ranking yet.

K3 for reference had 0.43 similarity to OpenAI's models, now it's as similar as how OpenAI's models are to each other at 0.58, much higher than other cross-lab similarities.

- Metaphor overload. We get it, it's just like car racing. Show some mercy on human readers.

- X, not Y

- A, never B

- Tasteless em dashes

- Hallucinated data, like model add date. The standings are so unbelievable that they border on laughable.

I ask it to plan the feature in a separate worktree.

In parallel I start coding without being biased by AI and vice versa.

At some point I read its plan and iterate on it all the while I am in implementation mode. This helps me improve my own vision.

Finally I ask the AI to review my implementation. It flags off bugs and gaps which are usually straightforward for it to fix.

For example model shares

<tool_call name="getWeather"> <param name="city">London</param> </tool_call>

Found the time traveller.

UIs it's generating is pretty good, not without problems, but certainly better than other models at this price point.

For example it's just so natural to share screenshots in a chat.

On my tasks it has not been as good as even Sonnet 4.6 so far.

Instruction following over long context feels worse.

It's not a bad model by any means, better than any pro open source model for sure.

Also as a person developing agentic code tools since before Claude Code, I'm skeptical if str_replace provides accuracy improvement over just full rewrite.

Back in the day when SOTA models would do lazy coding like `// ... rest of the code ...`, full rewrite wasn't easy. Search/replace was fast, efficient and without the lazy coding. However, it came with slight accuracy drop.

Today that accuracy drop might be minimal/absent, but I'm not sure if it could lead to improvements like preventing doc corruption.

My hypothesis is that some models err towards assuming human queries are real and consistent and not out there to break them.

This comes in real handy in coding agents because queries are sometimes gibberish till the models actually fetch the code files, then they make sense. Asking clarification immediately breaks agentic flows.

With search-replace you could work on separate part of a file independently with the LLM. Not to mention with each edit all lines below are shifted so you now need to provide LLM with the whole content.

Have you tested followup edits on the same files?

I couldn't decide on folding and reducing noise so I'm stuck on that front. I believe there is some elegant solution that I'm missing, hope to see your take.

GPT models can follow tool format correctly but don't keep on going.

Grok-4+ are decent but with issues in longer chats.

Kimi 2.5 has issues with it reverting to its RL tool format.

In my personal benchmark it's bad. So far the benchmark has been a really good indicator of instruction following and agentic behaviour in general.

To those who are curious, the benchmark is just the ability of model to follow a custom tool calling format. I ask it to using coding tasks using chat.md [1] + mcps. And so far it's just not able to follow it at all.

[1] https://github.com/rusiaaman/chat.md

This, along with John Ousterhout's talk [1] on deep interfaces was transformational for me. And this is coming from a guy who codes in python, so lots of transferable learnings.

[1] https://www.youtube.com/watch?v=bmSAYlu0NcY

TBF, Cursor's code indexing works the same way, it has to send all workspace files to their servers.

Auto-completion systems need previous edits to suggest next edits so no surprises their either.

from anthropic.types import MessageParam

data: list[MessageParam] = [{"role": "user", "content": [{"type": "text", "text": ""}]}]

```

This for example works both in mypy and pyright. (Also autocompletion of typedict keys / literals from pylance is missing)

It's fast too as promised.

However, it doesn't work well with TypedDicts and that's a show-stopper for us. Hoping to see that support soon.

- Cleanup or deletion tasks. Be ready to hit ctrl c anytime. Led to disastrous nukes in two reddit threads.

- Errors impacting the whole repo, especially those that are difficult to solve. In such cases if it decides to reset and redo, it may remove sensitive paths as well.

It removed my repo once because "it had multiple problems and was better to it write from scratch".

- Any weird behavior, "this doesn't seem right", "looks like shell isn't working correctly" indicative of application bug. It might employ dangerous workarounds.