Automated Unit Test Improvement Using Large Language Models at Meta

Re:Automated Unit Test Improvement Using Large Language Mode... by vasco on Saturday February 17, 2024 @03:22AM (#39407536)

What tells you something is how many releases get rolled back as a % and how much rework you have. If your developers need 3 attempts to put something working in production that's all you need to know about the quality of the reviews and tests. High or low coverage, what you need to look at is actual issues.

All you need is a bit hyperbolic, because you also need to quarantine flaky tests and other things, but coverage as a whole I think is useful only if you have an engineering organization that doesn't see the point in tests - but that's going to be an uphill battle.

[ Parent | on HN ]

Re:Automated Unit Test Improvement Using Large Language Mode... by polynomial on Saturday February 17, 2024 @10:26AM (#39410280)

Good heavens, a team needing 3 tries to get something into production sounds like they need a QA refresher.

[ Parent | on HN ]

Re:Automated Unit Test Improvement Using Large Language Mode... by withinboredom on Saturday February 17, 2024 @10:50AM (#39410536)

Most of the time, I see teams mixing refactoring, bug fixing, and new features into a single PR which causes this. Keeping PRs focused makes it easier to review and find issues before shipping to prod.

In other words, first bug-fix PR should be hacky af to fix the bug. No refactoring, nothing controversial (other than the hacky af fix). After you verify the fix in production, then, and only then, do you open a PR to refactor the code. Finally, after that is verified in production, close the bug ticket.

[ Parent | on HN ]

Re:Automated Unit Test Improvement Using Large Language Mode... by sdiupIGPWEfh on Sunday February 18, 2024 @06:25PM (#39424604)

I loved having small, focused PRs. So much easier to review without interrupting my day, too. Then we got handed a policy of one PR per ticket to simplify our QA process. Oh well.

[ Parent | on HN ]

Re:Automated Unit Test Improvement Using Large Language Mode... by nimblegorilla on Saturday February 17, 2024 @11:36AM (#39410967)

> Coverage as a whole I think is useful only if you have an engineering organization that doesn't see the point in tests - which is going to be its own uphill battle.

I think coverage stats are always useful as they help find the edge cases that people forgot to test. A common culprit I've seen is error handling code where a bunch of tests target the happy path, but nothing tests the error logging when something breaks.

[ Parent | on HN ]