Do most people generate PRs from a one-shot prompt? I have a whole pipeline of prompts, tests and evals, which loop back and potentially restart if the result is inadequate. How would I include that in the PR?

Would have to include the full conversion history. But who is gonna review that it's probably as big as some of these big PRs are in the first place.

Yep true, I'm still thinking about this as of today so my thoughts on this are not really mature and there are definitely a lot of caveats in that approach I didn't see.

However, having a rigorous spec process which ends up giving lots of details to the agent seems to me the right way to build stuff with LLMs. Rather than iterating and piling changes after changes.

Good question, and I think that this is a problem that needs to be solved. Just like a commit repository, it would be lit to record the chain of thoughts of the agents you're using.

At least for my workflow, which is fairly similar, the result of the first few stages is a refined single prompt, which I then run and the result is a PR. so I, at least, would include that final single prompt over the prior intermediate results.

I also think that might not be everything you need to really vet/recreate those PRs since there's a lot of strategic and operational stuff that goes into the pre-composition runs (sometimes I actually throw everything away and restart), so something a little more sophisticated would be interesting.

> Do most people generate PRs from a one-shot prompt

Obviously highly dependant on the complexity of the app.

Small CLI tools or helper react apps you'll throw away? Yes, I'll do PRs from one shots.

How does reviewing the long discussion the author had with the LLM prove the code is correct?

Because, unless you are changing some code by hand. Your prompt and the context your giving to your agent is the seed for the outcome of the agent itself. No agent as far as I know can generate content ex-nihilo. So having a look at what people are telling their agent to do can definitely help understanding the result of the same agent in my opinion.

If a PR is gigantic it means the team lead screwed up slicing the work.

Yeah true, actually my phrasing was poor. I came to that idea through debate with other coworkers. This is not a solution for gigantic PRs but rather a method on how people should be reviewing AI changes.

Why, yes, blame someone else, and not the person that shat out the PR :)

I had a recent case where I, being the lead, sliced up the work real good into an epic and what not and a guy on the team just did a "hurr durr, Claude, go wild with all that" and threw a huge PR over the wall linked to 5+ stories.

Well, we got to have "the conversation" after that and learned how to fix these kinds of PRs together...

It also means the engineer didn't check in with the lead as their PR became unwieldy.

That is patently false in the real world we live in currently. I constantly see lazy coworkers post 2000 line PRs for a trivial change that their LLM generated when 500 lines might accomplish the same in a better way. It's almost always a lazy human accepting what Claude did by default without any manual review or asking the LLM to do anything differently.

Claude absolutely loves to reimplement the wheel locally instead of just calling a method in a library we already depend on. Every doc comment and README.md update is bloated by 2-5x what's needed to communicate intent or to actually be read by a human later. It makes random drive-by edits that aren't necessary and weren't asked for in the prompt. It's a well oiled machine for producing astronomical bloat by default if you don't supervise and rein it in!

Just ban large prs. They can always be split up, and usually as well as helping review it makes them less risky and easier to find bugs and roll back if there was a problem.

The discipline of splitting them up helps in almost every dimension.

So what do you think of Discord rewriting in rust, and AWS shipping an emulator that was ai generated? Do you think those were done with small prs? Do you think that it's bad they did those things?

Please add your own thoughts to the conversation rather than aggressively asking rhetorical questions. "Counterexamples exist" is hardly a valid criticism of another human's perspective.

If someone says PRs need to be small, and I say here are some examples of huge PRs that I think are good, do you think these are still bad? That is not remotely invalid.

Where did grey-area say they need to be small. As in an absolute? They didn't, you just assumed that's what they meant and then went off the rails in your response.

The first 4 words? "Just ban large prs." A ban is pretty absolute language. He would have banned my example PRs clearly. Or maybe he wouldn't which is why I asked about them as examples. Because that is the obvious follow up. If you ban large PRs what are you losing? You're loosing big swing aggressive efforts to make sweeping progress.

[delayed]

It's like responding to "you should wear a seatbelt, statistically they keep you safer in a crash" with "here are two people who got in a crash without a seatbelt and survived"

I think they could do those things with small PRs if they chose. If their agents are not intelligent enough to make small prs, I guess they can YOLO and assume everything is just fine with what the agents did. I do think small PRs would be better for agent review as well though, because they avoid any problems with context being overwhelmed and things being missed/forgotten.

What are the PRs you're thinking of?

You're welcome to do large PRs, I'm not a fan of huge PRs though and would not pretend to review them (as pretending is what it would be), nor would I like to work somewhere where this is the culture.

I, personally, would certainly argue that those should be done in small PRs. If you have a working application and want to rewrite in rust, I would expect it to be done by replacing individual components one at a time while making sure relevant tests keep passing. If you're making a new application, I would expect it to be built incrementally by adding parts incrementally.

Yeah, if you ended up doing it as a single massive PR how can you ever be sure that existing functionality was preserved?

Discord's rewrite took 5+ years. Yes they probably used small PRs

Fully agree with a twist - ban dumb large PRs.

To add to my openspec shilling above, I've been known to "fix" large PRs by reverse engineering specs from them and building each spec out in a separate PR.

Last week me and a coworker turned a 15k line PR into 10 x 1.5k line ones that went in smoothly.

Yes I do like to make a big pr first sometimes but then it's usually pretty easy to split it into small ones and it's a helpful process in many ways.

Look into [openspec](https://openspec.dev). My team is routinely passing 1-2-5k line PRs through review, with the following caveats:

1. PRs come with "the prompt". In openspec parlance this is a set of docs: a proposal, design, specs, task list. 2. I have a tiny skill to slice and dice PRs, so I can review each part of the system separately by rewriting commits in order of reviewal: specs/docs, production code, tests, evidence. I sometimes pull DB queries to a separate commit for focused review. 3. The evidence bit is built by another skill: go over all added spec scenarios and produce a markdown document with links to the code implementing the scenario and the tests proving the implementation works. Emacs helps me jump back and forth via those links and read the parts I'm interested in.

My next goal is to extend openspec schemas with custom workflow steps to make PRs land in my preferred review structure and contain the evidence report, so I don't have to build it.

Your workflow looks really neats, I'll definitely give it a try.

So now I get to review the prompts and a slopped PR-description? Oh man, that sounds nice.

Happy to hear how you do it. We're all figuring that out.

Up until you notice that in the spec an agent wrote "All operations should be atomic". But, the you observe that it keeps modifying the same value across multiple threads because there's another spec that states "No perf regression, pls".

[delayed]