dhorthy

thanks. We have done a ton of red team pentesting with agents, both internally and through trusted partners. We're doing more continuously as we go.

yes, crazy to think that the original RPI hype was just over a year ago.

or someone could clone the SCB repo and make a fork that proves it one way or another

yep agree

they yearn to create content, generate more tokens, etc etc - agree

Yeah and I started with subsets although I did not find them representative, eg I picked 6 challenges for a subset and fable scored 33%, while on the full challenge it landed closer to 14-15% range

The part II post linked in my top comment has a list of the challenges I used for that.

To run them, just point your model at the paper, it can find the repo, etc and with a key or sub, get it running

great post - the same thing that has always been true of software engineering is still true:

Do fewer things in parallel, do them very well, optimize for getting incremental value out to users, FOCUS

It's been 30 years since the XP and Lean folks said "limit work in progress (50+ if you count the eli goldratt thing) - its still true

> AI-written emails where the commenter would rather see the prompt

this a thousand times this

LLMs are information transformers. if you're trying to take some rough idea and blow it out into something that "feels" substantive, you're just combining your high-signal idea / prompt with a bunch of meaningless bloat and noise from the model weights.

they seem to be able to do the call-stack diff stuff pretty well still

so wait is the finding that most of those skills reduce pass rates against SCB? wild

why would that worry you?

agree, i think the implication is that low quality code is harder to change in the future

somebody get this man a curl-pipe-bash stat

oh i really like the idea of flipping around the order of checkpoints and comparing results. Could be an interesting way to increase/decrease difficulty even

i will look into how easy it would be to zip up some subset of the results without leaking anything...probably doable

Yeah I would hold that models don’t know how to simplify because most rl/benchmarks doesn’t penalize complexity

Yeah the main reason I skipped fable was because we have a ZDR with anthropic and I didn’t feel like spinning up another account to circumvent that. Next run will have fable and sol

my issue with frontier code is that it uses a model judge for quality whereas slop code bench forces a model to grapple with its own garbage code in order to receive a functionality reward

i laughed at the pelican bit its good

yes the labs will always prioritize the vibeslop dopamine casino as far as I can tell - making the models useful and addictive for unsophisticated users, sometimes at the expense or at the very least at the ignorance of the needs of power users

yeah someone will have to re-run this bench on various effort levels. unfortunately it is not cheap

I agree this is an option, and the next thing on my radar is to try with a more realistic "factory-shaped" harness where you have feedback from linters and other models after each coding episode that refines the architecture.

For readability specifically, I've found it hard to get the models to do this with prompting. If you've talked to opus/fable for a long time on prose writing you probably felt this too

> - what "maintainable" is is probably some high dimensional space described by these signals; it'd probably require some human labeling to figure out where this space is

this is a nicely succinct way to put this - a multi-dimensional space where no single metric is really useful

state space of the system is interesting too. I would guess that for any production software with dependencies like databases/third parties that might be too hard to measure, but if you can silo off parts of your system into bounded state machines, it may be a value metric on some module behind a clean interface.

I think the kubernetes control loop model is a great instance of this, a handful of scoped components that own a control loop across a well-defined state machine, that can operate / recover in the face of most network partitions or downtime - the promise of CRDTs but rather more a pragmatic approach to it

yes sol is still my daily driver for most coding tasks

I did find opus 5 quite handy for general knowledge work and visual design, without the cost of fable (e.g. the graphics in this post are made by opus 5)

but its not noticeably better than opus 4.8 in those regards, and I would not miss it if forced to go back to 4.8