springtimesun

2 turns requesting information and indicating a preference for web search vs answering from memory and it fell over. Thinking block was a mix of Chinese and English, which is fine, but both questions were in English. Second turn answered in Chinese. When told “I don’t speak Chinese” it started to answer the first turn again, then crashed.

I find the benchmark results simply unfathomable after watching that performance.

Is the increase in commits graph in percentages because it’s fewer than Anthropic’s rate? A case of commit envy?

In all seriousness though I have enjoyed reading these posts. I will never work at this scale, but I like to read how different teams approach problems like this and I do wonder if some of these approaches will trickle down (or up?) into regular practice as everyone is committing more and faster.

I don’t know how many are still here, but do people who launched the project ever come back and get to give them a command now and then?

<thinking>

The voice recordings and non-linearity were what made it for me. I felt like I was piecing the world together as I explored. Even after doing the boss fight finding intel that explained something that wasn’t known before slotted it into my understanding of the story. By the time I reached the grind at the end, it was like I had a world in my head and it was ok to close the book.

I loved The Division. Some months ago I re-downloaded it and just ran around for a bit. The setting and world feel are still there and the story is deeper than some single player games. For a while I thought, wouldn't it be amazing if it was ever released (or leaked) so it could be player hosted or modded in a local kind of development like this. But, I didn't stay with that thought. There was just something about that game that had a beginning, middle and end for me. Maybe I'm just getting old.

I feel like I’m an outlier based on all these comments. This is still working through my personal benchmarks, but it’s spending 30% more time and performing worse than the previous 4.1 flash did.

Benchmark is historical PR reviews. Models are put in a sandbox with a git and db snapshot as it existed at the time. PRs are a selection of changes some bugs (major and minor), some with things that aren’t bugs that have tripped up models in the past and some no findings that they should approve and move along.

4.1 flash already wasn’t that strong at this task, but the speed edged out v4 pro for a spot in the roster.

I made a comment some time back that reviewing AI code was kind of bewildering (really any other code but my own). I’ve since dug in, crafted some processes and skills (in the /skill sense) and this is basically the shape it takes. Find the hot spots, what’s setting precedents, what’s not sitting right with me… it’s fast but it actually creates additional work (in a good way) because the way I have the skill set up the agent then scans through the whole codebase to find places where the changes demanded on a given PR also exist. Each PR spins out 1-multiple tickets. In a way, this has always been my dream. I believe in refactoring at some core level. Now I finally have the time (and tokens) to do it continuously.

LLMs have also pushed me to learn a lot, but I don’t find the shape to be terribly different than before. I rarely remember the exact sequence of commands or flags or syntax. What my learning was useful for before and after LLMs is to 1) build an intuition of how something behaves and what it should do and 2) expose me to non-obvious possibilities that I wouldn’t have come to on my own. With those two things in place, the next execution without the LLM is a docs search, same as it always was (for me maybe some people are better at remembering this stuff).

What’s different after LLMs for me is that there are some things that just don’t care about and I’m fine if they are a black box that the agent handles for me. I didn’t have that luxury before.

My take on picks and shovels was always a more cynical one. The beauty of the metaphor comes from the fact that after the miners went bust and left their picks and shovels where they lay, they could be collected and sold again to the next dreamer.

Claude has gotten pretty sensitive to copyright. In addition to the no song lyrics that was posted in another article, I also find it will not generally transcribe podcasts (you have to play with model versions and even effort inside that).

The “for agents” button answers a question I have had for some time: what happens when the frameworks I use that are well represented in the training data go stale? For the work I do, agents hit APIs nearly flawlessly from memory, but if the new releases from open weights well dries up, that won’t always be the case as the models age. I like this approach to building skills rather than relying on memory.

Man you people are hard ones. OP had an exchange (as a local) with someone who was impactful in their life. Not to slight Antirez, but I don’t think people are ruining his walks daily to talk about in memory caching. If it was really 5 minutes and a photo (he presumably agreed to) I feel like you all are blowing this way out of proportion.

I remember the first time I ever met someone IRL from a big OSS project. I was a little star struck. Of course he didn’t remember merging my code, but I didn’t expect him to. It’s an experience I will always remember.

I encourage anyone who has doubt about this question to try an experiment. Find a problem you are about to use an agent for or save one. Then ask a non dev to prompt the AI agent of your choice to try to fix it. You can even find a technical user or someone who uses AI regularly for non coding work.

I have done this a few times now and it is very apparent what experience, and skill, bring to the table.

I feel like this is a marketing miss. If they had held their announcement until the model was released, I would have grabbed it and started running it through my benchmarks. It probably doesn’t get a place in the rotation based on their own description of its performance, but now the weights live on the server, I’m probably following them on HF and I will remember to check in every time I ls the models folder. With the announcement only, none of that happens and I’m likely to forget about this by the time it actually gets released.

The email harvest move just doesn’t fit where we/they are in the cycle. There are established players and a buffet of models to choose from (plus a ton of empty hype). The first move at this point for any new entrant should be to show, not tell. Even an API only release with the promise to open weight would be better (actually probably all around better since most people can’t run this locally).

I wish this lab and all the labs releasing the best. It’s a brutal landscape to sink millions of dollars into for a guaranteed “behind x model from a year ago” evaluation. But, I do believe there is genuine innovation left to uncover.

The way I remember it marketed, and the way I originally found it, was as a workaround to setting the default search on iOS. This was before I was conscious of AI and before I remember it being in any of Kagi’s products.

I ended up staying because the Adblock is good (search remains good). The browser is fine. It does have bugs and needs to be hard closed occasionally.

I agree on the assistant. It is very capable. I use the Kagi mcp with coding agents and I like the results, but they have done something with their assistant that makes it even more useful.

The sensor readings were more supposed to be analogous to your organs giving sensory feedback information. Fair enough that "awareness" maybe should have been moved later in the analogy chain.

IMO it's not difficult to construct a scenario that technically should meet the definition of consciousness. Here is my technically conscious, but doesn't pass the smell test, hypothetical:

* A chat session is begun with a script to create a "heartbeat". The heartbeat simply to poll some sensors in the box and report back every x minutes.

* The chat session responds to each heartbeat as it comes in. Most of them are the same with small differences in sensor measurements.

* Over a long enough timeline, given the variability in token distribution, one of these responses will create a novel response.

* Over a long enough timeline, one of these novel responses will cause an action, maybe a new question, maybe the agent writes another script

* Over a long enough timeline one of these actions will create an action that becomes causal, it will create a chain that sustains itself. It may not be interesting, but in a network attached box, you can imagine how it can become interesting very quickly.

That's constructed in a cellular/genetic analogy, but because context is building and not lost to each generation, I don't think this takes all that long.

What does that leave you with? A box that has "awareness" of it's surrounds via sensor readings, a "heartbeat" that is driving it until it's unplugged or stopped, an input/output feedback loop that might be predictable in narrow sense, but is wholly unpredictable in its end state (a "will", if you will), and a thing that has the opportunity (but not the requirement) to introspect about its limits and possibilities. It can even do something that I would say is indistinguishable from planning. It can have goals. If you give it an Artifactory message board it can build a community ;)

The feeling is significantly mitigated/changed if you have a UI that allows branching. Begin your prior conversation and at a profound inflection point branch and edit the prior message. It’s a fun experiment to see what edits trigger very different responses, but running a wide ranging chat based conversation like this at temp 1 in replay will already show you something interesting.

Claude's suggestion at the end made me actually lol.

> Rewrite it darker — same events, but in the voice of the agents' own chain-of-thought instead of Frog and Toad.

I have read a lot of frog and toad to my kids. This was beautifully executed. But, I want to read that version too.

My partner spent many years in big law and now sits across the table on the client side. The billable hour has been problematic for much longer than AI has been in the picture, but what many people forget (including sometimes the companies paying the bill) is that you often aren't really paying for the hours, you're paying the structural tax that has been established by the system. Hours imply work product, but sometimes the reason firms are paying outside counsel _at all_ isn't because their in-house team couldn't do it, it's because the market/courts have told them they need the liability transference granted by a signed piece of paper.

That said, the rest of the article is on point. AI billing analysis software + internal knowledge of how the sausage gets made has shaved literal millions off bills at this point. I wouldn't expect them to lose money in an absolute sense though. More like a balloon you squeeze from one end.

One thing I came here to say, but turn out not to be the case, was the large jump in associates salaries as an interesting possible driver. I found a handy tracking number that does inflation adjustment. Turns out when you index for inflation, the associate pay scales are more or less flat.

This is a question I have been circling around. The agents are good at Ruby/Rails. It’s probably a confluence of things, but whether the L in LLM benefits from Ruby’s emphasis on human readability is something I’d love to see a study on. Or more generally, how key is (human) language itself in the performance ceiling of LLMs. A machine specific coding language/framework/process could obviously be more efficient, but if language itself is the hard limit, those would be dead in the water with LLMs. Or put another way: if language wasn’t the ceiling, why didn’t we see machine first coding implementations before LLMs?

This matches my experience. My solo projects are now bigger than I ever would have accepted alone. The one I’m going to finish soon is one I actually refused a year ago because it was too big.

It’s the top of the chain that I wonder about. Agentic development is so fast (and in the process I’ve developed so good) that I’m thinking there is actually a maximum size a company doing this work could get. Beyond that top end size the management overhead is what’s limiting or the market size total. Maybe not, maybe mega dev houses will find other problems to solve, but I have a hard time imagining it.

I have this in a rules file. It respects it. The only thing that broke recently was when they added the session links. That needed an explicit additional rule. Sometimes rules don’t get added to worktrees but I haven’t had the patience to isolate why this only happens sometimes.

This is where I landed when a recent(ish) change broke symlinking for rules files. I didn’t like how hacky it was a first, but now I’m glad I did it. The same hook drives specific rules the non-Claudes get per repo.

I was in the same boat around March. I started with Claude and giving it really specific instructions about what I wanted. That turned in to more interactive planning sessions where after we nailed everything down I said, ok just do it and I reviewed. The more I used it the more I let go of some things and the more I held on others. Now I have adapted my entire approach to use both local and cloud agents. Start small. Find what fits you. Also try out different models. Closed source is very strong, but you may actually prefer a bit less capable model that you give more direction to.

Yes, I'm with you 100% on the setup. You can go as deep as you want down the hole. At some point I had ask myself what I was really after with the benchmarks and where the line of diminishing returns was (and more importantly, was I really getting generalizable results).

I'm currently using a 2 tier grading system, one pass mechanical (LLM graded) one pass human. The grading ranges from -1 to 0 for the mechanical pass and -1 to 1 for the human pass. A model that does everything competently passed with a score of 0. Non-pass gets a -1. For the mechanical pass that's it. For the human pass there's a chance to get an extra point for exceptional work. Accepting the subjectivity in the human pass was part of the "what am I doing this for" point from above.

The scores then don't get read left-to-right and summed, but rather vertically per task and category. The temp setting as another commenter noted means you can't say anything definitive about the quality from a single run. What I think you can tell is 1) per task, who is the outlier in either direction and investigate, 2) how a task is generally handled by the models and 3) in the aggregate, who is dropping and gaining points consistently and does that change per task type. In the end, it's high touch. It doesn't give you one number to cut or keep a model. It tells you where to look and where not to.

You're the second person running an evaluation business to express interest, maybe I should start one as well ;) I will be reading your blog posts to see if there's anything I can adopt. If I was to give you the objection I would pose if this was an MBA brainstorming sessions I would ask:

Who are you targeting? and if the answer is engineers, what's to stop them from using the LLM to do it themselves? The problem space exists in the easy/hard domain; easy to set up, hard to refine because of nuance and specifics. Moreover, an engineer whose job might be under pressure because of LLMs would be highly motivated and highly capable to create a testing and evaluation setup and "own" this part of the process since it can't be replaced by an LLM.

Personally, I think this space is good consulting territory. There is hard-won knowledge that is valuable and sale-able, but beyond the basic setup, everything has to be tailored. It's also going to be hard to get people into a product moat. Git and IDEs give customers (if they are engineers) all the tools they need for this work and they already know how to use them.

One last note of your comment about same model agents. This is definitely something I have noticed. It's probably structural, but models absolutely prefer their own outputs to others. You can also move them toward agreement or disagreement by telling them that their model (in another session) produced some output. Kimi in particular will dunk on other models and promote what it thinks is its own output. Kimi is also highly sensitive to being bench marked. I have a specific audit script that goes through thinking traces to see if the model realizes it's being tested. Kimi figures it out more than any other model and will dig big time to find other results if you don't completely isolate it. Claude gets pissy if it thinks you are trying to benchmark it. Fable has inserted several times "not under testing" and "not a competitor" when left to its own committing. Fable is also very good at setting up open weights models and testing them. They must have spent a lot of time going for thoroughness and scientific method in training.

Curiosity and anxiety.

I rebuilt my entire workflow around agents so the unease that the frontiers would change something (access, pricing, availability) and lock me out of that were high. Also why I spent way too much on hardware (at least that can be deducted). Now the whole stack could run in my house and I feel much better about the situation.

Once I got the testing going though it is worth it for its own pursuit. Building processes around the dev process and trying to get the best outcomes is at least as fun to me as actually delivering client code. For the first time in my tech career I feel like I’m in a place with no maps. No one has done my experiments yet. I have a custom quant of K3 at Q5 that lets me get 10 tok/s on a CPU inference box (admittedly you need a 72GB Blackwell also). As far as I can tell no one else has done this. It’s such an exciting time!

In open weights models on a similar metric I find this to be highly specific to role and model. Some models are just diggers by nature, but all will verify more if you tell them to. Have you tried running a Claude watcher with the same role as Sol to see if it verifies as much?

I built exactly this using my own codebases. The setup isn’t that complex; split the git history to just before the change, sandbox the agent with everything they will need at that commit and lightly modify rules so they don’t go searching outside the box. Then they get the same prompt (usually the ticket that began the work) and are graded against the accepted PR.

The thing that takes the most time is finding the examples. In my real dev flow it’s rarely ticket -> PR -> merge, things bounce around a lot more. So, even though the stated goal is to get away from one shots, that is basically the environment you have to set or else test for specific other outcomes (e.g. agent stopped and raised a question when it realized x).

It takes time to do, but I would really recommend it. Now I can push new open models through the batteries and see how they line up to past ones in a few days (I run them locally, it’s slow). It moves my sense of x model is good at y and bad at z to from vibes to a better heuristic (these still run at temp 1, heuristic is the correct way to think about outcomes IMO). It grounds it in your actual code and problem space.

My takeaway from my testing: in Rails or front end codebases, most models I test are competent and with a human in the loop they would accomplish their goal of getting to a mergeable PR. They are not as good as Claude and since I pay subsidized rates via subscription Claude still gets first pass. They are very worthwhile to layer in as reviewers and catch many issues. My anxiety about a rug pull by the frontiers has been turned way down. I would have to adapt to a local only flow, but it wouldn’t be much adaptation and the opens can deliver in their current state.