The fact that tests are passing is not a useful metric to me - it’s easy to write low quality passing tests.
I may be biased, I am working with a codebase written in copilot and I have seen tests that check if dictionary literals have the value that was entered in them, or that the functions with a certain type indeed return objects of that type.
We should have two distinct, independent LLMs: one generates the code, the other one the tests.
perfect use case for GANs (generative adversarial network, consisting of (at least) a generator and a discriminator / judge) isn't it? (iiuc)
Do you also hire two different kinds of programmers - one that never wrote a test in their life, and is not allowed to write anything other than production code, and second that never ever wrote anything other than tests, and is only ever allowed to write tests?
It makes no sense to have "two distinct, independent LLMs" - two general-purpose tools - to do what is the same task. It might make sense to have two separate prompts.
Passing tests lol. Not my experience at all with Java
Describe what you want, get it to confirm a plan, ask it to start, and go make coffee.
Come back 5min later to 1k lines of code plus tests that are passing and ready for review (in a nice code review / diff inline interface).
(I've not used copilot since ~October, no idea if it now does this, but suspect not)