kvptkr

Ooh, interesting - starred and going to dig into this later today!

Interesting! I think you’re right in saying that the middleman you’re talking about has to be really good for something like this to actually be useful, especially for people very comfortable with debugging tools + their codebase. If I understand correctly, you’re saying that the most productive tool for you would be one that can present you with more relevant data, in a structured way (Redux, etc). At first, we actually did think of making a nice IDE with all that data, but found out it kind of already exists - https://pytrace.com/, and we found it to be more cumbersome to use than anything! Our belief is that these tools can help, but there’s an irreducible amount of reasoning that needs to happen, which takes time and effort, and we think a tool like this might be able to reduce that by offloading the reasoning to LLMs. I guess what I’m saying is that there’s a cap on how much useful information a tool can give a developer to enable them to reason better, and I’m really interested in seeing if it’s possible to reduce how much reasoning is needed in the first place. Curious to hear what you think - thanks for the thoughtful comment!

Oof, I'm sorry to hear that - I don't think we had any Django projects in the set of projects we were testing this out on. I just filed an issue here and hopefully fix it asap - https://github.com/leapingio/leaping/issues/2

So interestingly enough, we first tried letting GPT interact with pdb, through just a set of directed prompts, but we found that it kept hallucinating commands, not responding with the correct syntax and really struggling with line numbers. That's why we pivoted to just getting all the relevant data upfront GPT could need and letting GPT synthesize that data into a singular root cause.

I think we're going to explore the local model approach though - you raise some really great points about having more granular control over the state of the model.

Haha thanks! Yeah, I think that's definitely a logical next step. We do something similar for the larger bug resolution platform we've been working on, so it shouldn't be too hard to port over!