timabdulla
I would love for them to eliminate these issues because just touting benchmark scores isn't enough.
API downtime is a semi-frequent occurrence, as are transient API errors and slowness.
I've also had a ticket open with support for weeks due to rampant billing issues. For instance, a destroyed instance still shows up in my usage report as actively accruing billed time, and at a rate faster than is even possible (something like 2 hours for every 1 actual hour that has passed.)
They've released two new products in the AI space, this and Phoenix.new, and my worry is that they are focused on new products over making what they have good and reliable.
Can you comment on the variance? It's impressive that models are able to do this consistently with 100% accuracy in the early months, but it would be less so if there was any significant degree of variance amongst the three runs (e.g. 90%, 95%, 100%.)
If autonomous agents were just around the corner, then why wouldn't OpenAI bet on their own Codex product obviating (most) need for an IDE and save themselves the $3 billion?
It obviously knows the game through and through. Yet even with encyclopedic knowledge of the game, it's still a struggle for it to play. Imagine giving it a gave of which it knows nothing at all.
[0] https://claude.site/artifacts/d127c740-b0ab-43ba-af32-3402e6...
Definitely can be very annoying if you do just want it to execute on a set of instructions.
It didn't pick up my HN comments, probably because my first and last name are not in my profile, though obviously that is my handle in a smooshed-together form.
The first was on a topic with which I am deeply familiar -- myself -- and it made three factual errors in a 500-word report: https://news.ycombinator.com/item?id=42916899
The second was a task to do an industry analysis on a space in which I worked for about ten years. I think its overall synthesis was good (it accorded with my understanding of the space), but there were a number of errors in the statistics and supporting evidence it compiled, based upon my random review of the source material.
I think the product is cool and will definitely be helpful, but I would still recommend verifying its outputs. I think the process of verification is less time-consuming than the process of researching and writing, so that is likely an acceptable compromise in many cases.
It stated that I had 47,000 reputation points on Stack Overflow -- quite a surprise to me, given my minimal activity on Stack Overflow over the years. I popped over to the link it had cited (my profile on Stack Overflow) and it seems it confused my number of people reached (47k) with my reputation, a sadly paltry 525.
Then it cited an answer I gave on Stack Overflow on the topic of monkey-patching in PHP, using this as evidence for my technical expertise. Turns out that about 15 years ago, I _asked_ a question on this topic, but the answer was submitted by someone else. Looks like I don't have much expertise, after all.
Finally, it found a gem of a quote from an interview I gave. Or wait, that was my brother! Confusingly, we founded a company together, and we were both mentioned in the same article, but he was the interviewee, not I.
I would say it's decent enough for a springboard, but you should definitely treat the output with caution and follow the links provided to make sure everything is accurate.
In WebArena, Operator does 58.1%. Previous SOTA for browser-use agents is 57.1%. In WebVoyager, Operator does 87.0%. Previous SOTA for browser-use agents is the exact same.
See here for details: https://openai.com/index/computer-using-agent/
The truth is that while 87% on WebVoyager is impressive, most of the tasks are quite simple. I've played with some browse-use agents that are SOTA and they can still get very easily confused with more complex tasks or unfamiliar interfaces.
You can see some of the examples in OpenAI's blog post. They need to quite carefully write the prompts in some instances to get the thing to work. The truth is that needing to iterate to get the prompt just right really negates a lot of the value of delegating a one-off task to an agent.
To me, it's a lot simpler. I use forms and links and render HTML templates. That's it.
I don't need to worry about building and maintaining a JSON API just to serve the front-end. I don't need to worry about distinct sets of routes and models. There's just one repo with one build pipeline and test setup. There's no need to worry about front-end state at all, since I'm just using plain-old links and forms.
I'm not claiming that there is never a use for React or its ilk, but I am genuinely finding it hard to understand how one can assert that this simple HTML-based approach is categorically worse in all circumstances.
In my context, the simple approach works. If I were working on something that had a high degree of inherent complexity on the front-end -- maybe a web-based video editor or some kind of WSIWYG productivity tool -- this approach would almost surely fall apart, but that's the point of engineering: Choosing the right tool for the job at hand.
I read "drop-in remote worker" as AGI. You can give it any task and it performs at-or-exceeding human level. The real question to me then becomes the implications for the rest of the economy, not simply a question of what happens to vertical AI companies.
So many B2B tech companies exist to make the work and organization of humans easier. Is Github still as valuable if the vast majority of code is written and reviewed by AI? What about Slack, Linear, or Salesforce? And that's just starting with tech.
If there are relatively few humans in the white-collar workforce, then we are talking about nothing short of a complete remake of the economy.
In my opinion, the article spills a lot of ink trying to prove something that to me feels obvious (given that you accept the premise that one day soon we will have AGI) and very little exploring implications beyond this narrow perspective. Perhaps that is coming in future chapters.
I think with a lot of AI folk in San Francisco, this is a tacit assumption when having these sorts of conversations.
If a general AI model is a "drop-in remote worker", then UX matters not at all, of course. I would interact with such a system in the same way I would one of my colleagues and I would also give a high level of trust to such a system.
If the system still requires human supervision or works to augment a human worker's work (rather than replace it), then a specific tailored user interface can be very valuable, even if the product is mostly just a wrapper of an off-the-shelf model.
After all, many SaaS products could be built on top of a general CRM or ERP, yet we often find a vertical-focused UX has a lot to offer. You can see this in the AI space with a product like Julius.
The article seems to assume that most of the value brought by AI startups right now is adding domain-specific reliability, but I think there's plenty of room to build great experiences atop general models that will bring enduring value.
If and when we reach AGI (the drop-in remote worker referenced in the article), then I personally don't see how the vast majorities of companies - software and others - are relevant at all. That just seems like a different discussion, not one of business strategy.
I think it's obvious that they've cracked the formula for solving well-defined, small-in-scope problems at a superhuman level. That's an amazing thing.
To me, it's less obvious that this implies that they will in short order with just more training data be able to solve ambiguous, large-in-scope problems at even just a skilled human level.
There are far more paths to consider, much more context to use, and in an RL setting, the rewards are much more ambiguously defined.
I believe about 90% of the tasks were estimated by humans to take less than one hour to solve, so we aren't talking about very complex problems, and to boot, the contamination factor is huge: o3 (or any big model) will have in-depth knowledge of the internals of these projects, and often even know about the individual issues themselves (e.g. you can say what was Github issue #4145 in project foo, and there's a decent chance it can tell you exactly what the issue was about!)
There's a difference between minification and obfuscation, but again, I'm not sure how they adjudicate it or how much they enforce it.