Prod your agent first until it can explain back to you what's wrong with the tests.
Then ask your agent to write a better prompt for writing tests (and save it to an .md file)
Then reference that .md file and iterate on the tests again. It may take a couple of tries, but I've had good results with this strategy.
Ended up with a whole prompting framework.
To be fair, some people like to prod. But, for me, I'd rather take hours reading manuals than do trial and error prodding for 30 minutes (even if I knew it would be that short). Yes it's not optimal in terms of time, but it's better for my soul and well being.
It is also optimal in terms of time in the long run. The prodder will have to prod over and over again because he has learned nothing. You on the other hand will actually understand what is going on and will be able to either do it yourself or give precise instructions to the machine next time. Don't let yourself become a meat proxy.
Maybe that's not a big deal, they can still be generally useful, but they seem important to commit with the project if they will be used going forward. I hate the idea of a massive amount of slop riding along with the code, and slop begets slop.
The idea I'm chasing now is writing the documents myself with the hope that enough examples and a document about writing documents will be enough to allow it to write clearly.
It is the same like humans: if they say your code is wrong, you will hire someone else who doesn't find so many bugs. It doesn't look good.
> Of course, core developers don't need to check this monotomous checklist. This is only useful for the newcomers.
So this is saying core devs already know not to use AI, only new developers shouldn't.
It's quaint at this point. Embrace the future.
Having projects with properly scaffolded code, et al, will be valuable when the FAA grounds all planes for a week because of <insert all the grusome things you know this administration turns a blind eye to>.
Or is it just me?
You have to serve users, not hold code up as a sacred relic.
Our own DNA was vibe coded. It's a mess. Stochastic hill climbing can get pretty far, and it can reach surprising fitness.
Our food supply has perfectly healthy options. People choose to buy and eat junk, then blow up to 400lbs. You can gild the lily by arguing over organic vs Amish-grown in terms of micronutrients and traces of pesticides, but even an inorganic ear of corn is 10x healthier than what the average American eats.
The people looking at AI and taking a "none for me thanks" stance are looking at externalities and saying they don't like them, and they dont like them enough to not participate.
Personally I consider it bizarre because the entire thesis and history of computing, going back to the term "computer" referring to a human who did tedious arithmetic by hand, was leveraging systems to accomplish a goal. Except now. Because now for (often) vague purity reasons, we should no longer reach for that lever.
That "vague purity reason" is almost always software developers so engrossed in their art form that they are more interested in the code than the product. There's not necessarily anything wrong with that, but if your goal is to ship a product, choosing not to reach for a lever to make the product better meets my definition of "bizarre".
It's bizarre in the same sense that abstract art is be bizarre. It can be interesting, novel, and even thought provoking; but often lacks functionality or utility to make it applicable to the lives of people. That's what non-LLM projects going forward will be.
If you start with the assumption that AI is a tool, then finding uses for the tool is logical.
Seems to me that some people really love riding a purity spiral, and collectively they have chosen a new tool to fill the role of the impure thing to be cast out.
Someone can believe AI was trained on material in a way that violated the consent of authors and artists and musicians. Someone can see the value of the tool, but not want that value to be primarily captured by megacorps. One can believe that the tool is having a deleterious effect on students, workers, and artists of all kinds, as it's being deployed. One can't be concerned about the added environmental effects, the impact to world markets, or the shortages it causes to consumer hardware. One can be worried about the copyright implications (can AI produce copyrighted material?) and not want to risk losing their IP. One can believe that art is necessarily a human enterprise, and art produced mechanically is just images and sounds, but not art nor music.
None of these are "the tool is the devil". All of these are "the tool has externalities, and those externalities come with a cost I am not comfortable paying".
I'm a vegetarian. Not because I think eating meat is immoral, but because I believe factory farming causes undue suffering to animals and the people who raise and process those animals. So I make a choice for myself, in my own life, and don't moralize to others.
It's the same thing. Some folks are making choices for their own lives based on their values for those externalities. You value energy use differently than they do. You value art differently than they do. You value corporate control over communications differently than they do. Neither of you is right, but standing here and pretending an eminently justifiable difference of opinion is just purity signaling is boorish.
They may be proud of not paying the externalities that the evil megacorps are demanding, but that warm blanket of purity is no protection from being outcompeted.
I think it's a real shame that so many people have set themselves against an emerging technology for reasons that on closer analysis seem like just another round of prejudice and superstition.
No, that was Reinforcement Learning
They are serving users, by not allowing AI slop. Software developed by people and not an LLM is of much higher quality, and I'm glad they are taking a stand.
Someone else is welcome to fork GIMP and use AI to serve the users, if they so like.
No one "has to" do anything.
I find that the human written tests on my project were the same. We were guarding not just the business behavior but also weird cases that have nothing to do with business, and also self-evident cases that basically boil down to x=x or assert(true).
Not to mention the stupid side-effect of not updating mocks when the APIs move but tests don't test that part so they are okay for a lot of time, and suddenly after a non-related change they are not.
Gimp has millions of downloads this month. They are doing something right.
Microsoft also seems to use "models" to "test" its programs. Look how shitty they are.
Does GIMP have a very restrictive AI policy or something? I get not wanting to ship a gigantic pile of UI tests without thorough review, but basically any model released in the last six months would be able to write UI tests covering every single control in an afternoon.