pjm331

great writer, starts forgetting instructions around 80k context in my experience

markdown specs are certainly the funniest way to claim to distribute software

I was demo'ing Gemini 4 Argon last night and watching it use codemode to basically generate structured docs about the codebase for itself

so your experience matches my own limited experience of starting to run evals of my own harness, which is that these specialized tools can help weak models but on the frontier they just make it slower and more expensive

I'm not looking to be convinced about this tool as much as I am genuinely looking for a way to eval these kinds of tools in general

I have only just begun playing around with evals for a harness I'm working on and so far I have just been doing runs of converting https://github.com/realworld-apps/realworld from one language to another

not super pleased with it

the unsatisfying answer I think is that crafting good eval environments for long runs takes a lot of time, and there are many things you'd want to verify that you cannot simply lint or test for

who out there is running evals of giving agents these kinds of tools and how are you doing it?

I have a project I am working on where I wanted to have nice docs, and so I have some slop docs sitting there as placeholders until I get around to writing them, awful claudish stuff

I've actually spent some effort trying to make them halfway decent placeholders by putting together a skill based on diataxis framework (https://diataxis.fr/) that at least got everything into the right location

just took a pass at rewriting them w/ argon and am very pleased, big improvement

I’m guessing they’re referring to jamal khashoggi

Yes I have just started dabbling in some harness evals where the task is translating the RealWorld “mother of all demos” app, maybe one of the most translated apps of all time, and they cannot reliably 100% it

I don’t write the same program over and over again so I am always confused by how exactly this industrialization metaphor is supposed to map on to software development

I appreciated this post, had never heard of codemode and I think the name is bad but it makes perfect sense

I have wanted something like this ever since I hooked up my first database MCP

Originally I had thought to set up a python runtime with like a standard data science toolkit and the db mcps available as functions, and maybe I still will but JS has been working fine for now

I’ve been slow blinking at every cat I meet for years, I swear by it haha

callback urls are all localhost, there’s no domain to whitelist, just client names

Another thing they do that I find equally frustrating is their MCP can do things you cannot do via API so you are forced to use theirs and cannot implement your own

this is how i've been solving the problem before some of these newer techniques around deferred loading etc. came out

a subagent per mcp toolset, basically sub agent per saas in the configuration at $WORK

I don't have any hard evals on this but it feels like it works more than it doesn't, especially now that subagent use has gotten better in general, but you do feel drawbacks if you ever need to do something that involves coordinating multiple sub agents, there's some communication overhead that wouldn't be there if it was just one agent with all the tools to do the whole job

yes codemode is when you want to utilize this ability AND you want MCP tool calls as part of your scripts

codemode lets you execute scripts in a runtime where your MCP tools are made available as function calls

this matters for cases where the MCP tool is the only way to do something and you do not have an equivalent CLI, API, whatever to script with

I’m not sure. my read of this was that AI weakens human ecosystems in general because we don’t need to work together as much when we are all just working with AI separately, but maybe you have specific examples of language agnostic ecosystems in mind? I’m struggling to imagine what that would look like

Didn’t mean any judgement just the metaphor that came to mind

Roblox is a game where you make the game. This sounds like an OS where you make the OS.

Probably could have spelled it out more, being too pithy perhaps

Adult Roblox

We are many of us now engaged in a big experiment to see just how little you can read the code, and for how long. Considering the consensus is that we have still not yet passed the 1 year anniversary of agents getting good, it feels too early to say how it will play out. But there are a lot of market forces working to make it happen.

> That is currently working, but since the fix is fully deranged I expect it’ll be disturbed again at some point.

This one made me laugh out loud

It’s all experiments at this point

Clear and simple as the truth - another good one

If I were you I would look into maybe like analyzing sessions across harnesses? I’m not sure, some way to leverage this shared schema work that is not syncing

Because syncing is trivially solved by just telling the agent to keep a work journal in markdown

yup just joined the waiting list with a very similar use case in mind

I’ve always been surprised about the lack of discussion of amp on the front page here.

Tough selling dev tools these days

It feels like the Venn diagram of people who would use orbs and people who will just implement orbs themselves on top of something like E2B is basically a circle

Frustrating because JJ does have workspaces which are the comparison you’d actually want to make https://www.jj-vcs.dev/latest/working-copy/#workspaces

miro is my go-to for putting together diagrams for slide decks, but after having tried using it to actually work on things a few times, i've just come to the conclusion that actually virtual whiteboards are a bad UX for most things, inevitably you spend more time organizing the board than actually getting work done

I need to understand what is the UX of departments. 3d animated wood panels on a forest floor??

This has been a nice thing about using doctests in elixir. doc examples are compiled and run so they can’t drift. The prose around the examples can still drift of course