bhackett

OK thanks, agents definitely like partial implementations like that. When this impacts users we need to be able to observe that and see the bug, then track it back to its source. Weird logic bugs like this benefit from having a good debugger to untangle things and get to the root of the problem, especially when there is tons of code.

If you're happy with your agent's performance using surface level o11y to track down issues then that's fine. This is what pretty much the entire industry is doing and it does usually work! QA focuses on root causes because it can look through the app's entire execution and provide evidence so your agent doesn't need to guess when figuring out what the fix is. That guesswork is what we're trying to eliminate, for when the surface level stuff doesn't clearly point at the problem.

What sort of bugs do you have in mind? From our perspective there are many kinds of bugs that inhibit UX (and AX) but they can be subtle issues like poorly designed flows. It's hard for an agent to notice or understand these but they are serious problems and we're trying to surface these more readily.

Agents that are blind to understanding the problem they created will definitely not be able to fix it. Our argument is that we can fix that blindness with better observability. For better or for worse, even if the code is messy, if an agent can see what is going on it can keep things together and get the app into working shape. 100% human written too btw...

I think AX is just as important in that case though, as even a patient agent will make mistakes and will have a lot more trouble getting things done right when using a buggy app.

There's an opportunity here to bend things towards quality and efficiency moreso than mere volume of code. The agent has to be able to effectively repair and maintain what it has been building though.

Convoluted code is definitely a problem, I don't think it's unique to AI though. It also becomes more manageable with a time travel debugger as the actual behavior is easy to understand.

(OP) I completely agree! AI is amazing at writing web apps that are full of bugs, but it also gives us what we need to turn that completely around. That's what we're trying to make happen...

Well, it can be both. This post https://blog.replay.io/the-nut-api is the best technical overview of the API and discusses several examples.

Yeah, we allow importing projects so you can fix bugs you've encountered elsewhere, but we want to streamline the app building process and the UX used in v0/Bolt etc is already really well done. We want this app building to be more reliable though, it's easy to hit bugs that prevent progression.

We're also interested in using our API with MCP so that e.g. Cursor could be used to fix bugs you're seeing locally, and plan to explore that angle before long.

Hmm, strange, it's loading alright for me but we've had a couple reports of rendering problems. If you have a chance to file an issue here https://github.com/replayio/nut.new/issues I'd appreciate it, thanks!

Yeah, the source is here: https://github.com/replayio/nut.new

We'll continue to keep it open source as we develop it.

Yes, that's all true. Even so, vibe coding empowers anyone who can write clear instructions to build software, but the limits of the technology get hit pretty quickly by non-developers and they have little recourse. This blog post https://addyo.substack.com/p/the-70-problem-hard-truths-abou... is a great overview.

The tech will get better and better (I couldn't imagine we'd be doing this a year ago) but to be truly useful it has to reliably produce reasonably well engineered code, and effective debugging is a key piece of that.

Thanks! These are both fixed now. Clearly we need to do some more dark mode testing...

Thanks, yeah we're really thankful to StackBlitz for open sourcing the early version of Bolt.new and to the Bolt.diy community for continuing to develop it.

We don't have a commercial offering yet and are planning to migrate off WebContainers for the upcoming full stack features -- WebContainers show their limits pretty quickly in a full stack context (e.g. CORS issues) and we need observability into the server side of the app for full stack debugging.

Regardless, our interests here are only lightly commercial. We're not really developing Nut to drive revenue but to help us develop the debugging API and push forward the SOTA for AI development as effectively as we can. That API is what we want to sell.

Yes, Nut is a fork of https://bolt.diy and like bolt.diy you can add your own API key and use it as much as you want (Nut is hosted though so you don't have to set anything else up).

The improvements we're making are under the hood. When you ask Nut to fix a bug it should do a much better job -- we record the app's behavior and analyze it so the AI has context for the changes it needs to make.

We've also added some UI to approve or reject the changes the AI makes. For now we're using this to gather feedback so we can improve Nut, but down the line we'll also refund the user any credits when they reject changes -- you shouldn't have to pay when the AI screws up, a big issue with these tools (and vibe coding in general).

Thanks! Unfortunately the chat links aren't shareable yet. We're planning on adding this within the next couple weeks along with the other full stack features (database integration and easy deployment).

Thanks for the report! The about page is fixed now when looking at it in dark mode.

We definitely need to avoid replaying from the start every time we want to inspect the state at some point. This is kind of an internal detail, but we can avoid having to replay parts of the recording over and over again by using fork() to create new processes at points within the recording.

Ordering constraints between different library functions do crop up from time to time. In cases like this the recorder library uses ordered locks internally (basically emulating the synchronization which the system library has to do) to ensure that the calls execute in the expected order when replaying.

(Replay employee)

1. Rather than having to restore state to the point at the previous step, we can step backwards by replaying a separate process to the point before the step, and looking at the state there (this post talks about how that works: https://medium.com/replay-io/inspecting-runtimes-caeca007a4b...). Because everything is deterministic it doesn't matter if we step around 10 times and use 10 different processes to look at the state at those points.

2. We record the calls made by the browser, though it is the calls into the system libraries rather than the syscalls themselves (the syscall interfaces aren't stable/documented on mac or windows).

3. Maintaining ordering like this isn't normally necessary for ensuring that behavior is the same when replaying. In the case of memory locations, the access made by thread 2 to location B will behave the same regardless of accesses made by thread 1 to location A, because the values stored in locations A and B are independent from one another.

(Replay employee) Yeah, later in the post (https://medium.com/replay-io/effective-determinism-54cc91f56...) we mention needing mitigations to ensure that hashtable iteration is deterministic in cases where it affects how the browser interacts with the JS engine. There are others needed as well to ensure that the browser behaves the same across a large range of websites, my favorite is handling sites that sniff information about the system's math libraries by e.g. calling Math.sin() on specific values and testing the results --- the values when replaying need to be bitwise identical with what happened while recording.