palashawas

Right - we did this benchmark because we launched a plugin that makes APIs programmatically from an app's human-oriented UI (from the event handlers, to be specific). So any app that has a human-oriented UI now has an API.

The benchmark is a more generally interesting part of the launch materials, so I figured it had its own separate home here.

This is a fair point.

The models frequently failed for many reasons on earlier runs, and the browser-use prompt ended up being pretty granular. I'll add a couple of runs that include a scroll instruction to the repo today and see how that compares

Pretty hard to guess what Anthropic trained sonnet on, but general multimodals are what people are using to drive similar tools today, whether GUI-trained or not, so the comparison still holds, for now

Interesting! I'll play around with agent-browser and update this article if anything comes up

Yep, everyone knows computer use is more expensive. This is about quantifying the gap