dennispi

Actually, not generated by AI. It was largely handwritten.. but since you found this phrasing objectionable, I removed it.

Agreed. While “Cost per task” is a much better unit than cost per token, it’s still an intermediate metric.

The metric we actually care about is closer to "cost per accepted outcome" which would roughly be "total inference + retries + failures + the human effort required to verify, correct, or resume the work."

The first few are easy to instrument.

But "human effort verification" and “accepted” are much harder to define consistently across different agent workflows.

For our code review product we have a decent proxy because we can measure user reactions and whether findings are actionable and if it's merged/accepted.

For more open-ended agent tasks, I don’t think the industry has a particularly good answer yet... I would love a point if there is one?

The cheaper model isn’t actually cheaper if a human has to spend the savings babysitting it...

We built something similar an A/B testing framework that measures Unblocked's impact on real AI coding agents.

It spawns agent CLIs (Claude Code, Codex, Cursor, GitHub Copilot) with and without Unblocked's MCP server attached, then statistically compares the results: https://github.com/unblocked/unblocked-harness-compare

We likewise measured token savings, (wall clock) time, # tool calls, and # turns.

Unblocked CEO here, happy to answer any questions!

As one of the Founders of buddybuild, I'd like to perhaps also provide another perspective on this specific thread.

We certainly don't think of buddybuild as a closed ecosystem.

In fact, our users are able to customize buddybuild infinitely to meet the needs of their workflow (details can be found here: https://www.buddybuild.com/blog/customizing-the-build-proces...)

Further, while buddybuild is favored by thousands of because it can can auto-provision iOS devices for you on the fly. Meaning, you never have to deal with provisioning profiles and UDIDs ever again.

We also care deeply about making sure that teams can involve and engage their end users as part of the development process. One of the reasons we built buddybuild is because of the frustration sending builds to our users... AND then never hearing back from them. Phones are great devices for "consuming content", but it became obvious that the physical form factor of the device tends to result in infrequent feedback which was often ambiguous.

So, we build an SDK that gives you high quality and actionable bug reports from beta users. With a simple screenshot, testers can file bug reports that include graphically annotated screenshots, feedback notes and their device metadata.

We also have a feature called "Instant Replay", which lets you watch a video of the exact steps required to reproduce a crash. Instant Replay works in conjunction with our Crash Reporting solution to offer you both the exact lines of code that caused a crash as well as the exact sequence of events that lead up to it (https://www.buddybuild.com/blog/introducing-instant-replay)

Thousands of teams with fairly complex needs use buddybuild every single day. They teams like Slack, Meetup, FireFox, Wordpress and others..

You can read why they made the choice to use buddybuild here: http://www.buddybuild.com/customers

Lol- pun intended? (...Excel)

Ah, I should clarify and provide context.

We're a mobile focused continuous integration and deployment solution.

When deciding which platforms to support, we looked at current adoption for Native iOS and Android (this was no brainer), Hybrid (PhoneGap, Ionic, Cordova) and "emerging" platforms (React Native, Xamarin).

Ultimately, we decided to prioritize the others first. CI/CD service is applicable to any mobile development team.. but to your point, buddybuild is better suited to those other platforms, as those are the only ones we support.

In our experience (which also factors inbound requests), Xamarin is still relatively nascent as compared to the others.

That said, you can should expect to see Xamarin support soon :)

Also, fwiw, Microsoft purchased Xamarin only recently.

At buddybuild, we see tens of thousands of mobile projects from mobile dev teams.

Of the teams looking for cross platform solutions (versus native iOS or Android), Xamarin comes up on occasion.

However, React Native tends to be FAR more common as teams are looking for solutions for "write once, run on many"

I can't think of a better, more appropriate Cohen cover: https://www.youtube.com/watch?v=AGRfJ6-qkr4