pistoriusp

Same, that percentage seems absurd though.

> Since any LLM will answer all those questions competently

That's false. The LLM will only answer competently if it was trained on that data; and if it has enough data to make the correct connections between your question and the "correct" answer.

In the case of this article they're specifically saying the LLM has limited training.

Aren't you missing OP's point entirely? Which is: If the LLM didn't have useful information it would still give you an answer... Helpful or not.

Which is why you should perform your own benchmarks against your own software stack.

I checked out the development benchmarks for agents, and was suprized to find that they're not really representative of many of my own day-to-day development workflows.

It appears that they're mostly testing the ability to make business tasks autonomous, with ~20% associated to development tasks (ssh here, install this, etc.), but not actual programming.

I think this is less about the actual use-case and more about the approach; so a comparison is moot.

Interesting. I had no idea that a person could see what is flagged?

I had the same experience, but someone calmed me down by saying "we're all working on solving this problem."

Apparently "/prewalk" is built into https://github.com/can1357/oh-my-pi, which I guess is a lot like "Oh My ZSH?" Pi with batteries included.

There are a lot of interesting ideas in it, mostly none that I have applied to my own workflows. Curious if anyone has used it?

You've got to place that within the context of Sub-Saharan Africa.

Mac Mini in the rack and a Neo in the lap.

Built something like this for the v1.0 launch of RedwoodSDK in March; but it's a popularity graph (like the time scrubber on YouTube). I figured that it could be useful to myself, and others, to aggregate the information that people find most useful.

https://rwsdk.com

Why not just run a vm?

Seen this a few dozen times before, they're just not capable.

Gonna reply guy here because this is a paid thing. Agree with the author. Exposé was amazing. Here's an open alternative that I built, completely keyboard driven: https://github.com/peterp/cmdcmd/

Please try agent-ci; it's github actions that run locally. Nothing less.

You're right, figured it was implied, but now fixed.

I did not say that, what I said was: It's not like `act` because it's not a rewrite of the runner. It's the standard runner... So the one that actually runs GitHub Actions.

I have tried to use act many times, and many times I've failed.

P.S. pause on failure is also helpful for humans, but I'm trying to be realistic about where the future of programming is going...

We support macOS via tartlet, but basically it's always linux. If you need windows then it's gonna be an issue.

The jobs runs via containers.

Whilst you're waiting for it to come back, try out AGENT-CI (which is a project I built.), which runs GitHub Actions on your machine: https://agent-ci.dev. (Open source, etc.)

No, it's not like "act," because it uses the standard Github runner, the difference is that the control plane is an emulation of api.github.com, because of this we can do all kinds of nice things:

Caching in ~0 ms. Pause on failure, so you can let your AI agent fix it and retry without pushing.

This got me thinking about code.google.com, I can't believe Google dropped the ball that hard.

if you run it at home then the sun is a pretty good way to get "free energy."

https://agent-ci.dev: Run GitHub Actions on your machine. Caching in ~0 ms. Pause on failure. Let your AI agent fix it and retry, without pushing!

Scott is brilliant, funny, and kind, and maybe he could be serious if he ever needed to be serious... But I've never seen that in him.

I don't use copilot, but somehow was subscribed... I probably clicked something long ago and it just remained active.

Pleasure! Yeah it's just the standard runner - the only thing we've built is a duplicate control-pane, which is just HTTP. Since I have the standard runners source code it's pretty trivial for agents to know what to implement on the other end.

Repo: https://github.com/redwoodjs/agent-ci, that includes a demo in the README.

Hi HN,

I built this because I treat CI as the last line of defense. Agents also need validation. They should use CI, and they shouldn't bother you unless everything is green!

GH Actions is usually in the top-5 expenses for dev-teams. Add agents to that mix? It'll easily double. It's the wrong tool for the right job: Slow boot, slow cache, retrieving logs is token expensive for agents, the list goes on...

So I built a tool with one amazing feature: live-reload for failures. Agent-CI is a local CI runner.

I tweaked the control pane and mounts to provide 0ms caching, insanely fast boots. When a step fails it pauses, provides the agent with the failure, and waits for the agent to fix and retry just that step.

It uses the standard GH Actions image (via Docker), but emulates the control pane via a local HTTP server. You don't have to change any of your existing GH workflows.