Someone vibe coded a PR on my team where there were hundreds of lines doing complex validation of an uploaded CSV file (which we only expected to have two columns) instead of just relying on Ruby's built-in CSV library (i.e. `CSV.parse` would have done everything the AI produced)

I wonder how many times the LLM randomly tried to steer back to that library only to get chastised for not following instructions.

That’s a good example of ‘getting a desired outcome based on prompt’ - use a built-in lib or not.

And when it hallucinates a non-existant library, what are the magic prompts that you give it that makes it stop trying to bullshit you?

> what are the magic prompts that you give it that makes it stop trying to bullshit you?

Maybe keep your eyes open? :-)

As I thought.

And for the record - my eyes are open. I'm aware I'm being bullshitted. I don't trust, I verify.

But I also don't have a magical lever that I can pull to make it stop hallucinating.

... and every time I ask if one exists, I get either crickets, or a response that doesn't answer the question.

Ask it to write tests, then let it run until the tests pass (preferably in a sandbox, far from your git credentials). It is quite good at developing hypotheses and tests for them, if that is what you explicitly ask for. It doesn’t have (much) ego, so it doesn’t care if it is proven wrong and will accept any outcome fairly if it is testable. Although sometimes it comes to the wrong conclusion and doubles down that the fact should be true so it prepares to write and publish a library to make it true

Sorry! Didn't mean to BS you. I've not come across a scenario where it hallucinated me with a non-existent library. Can you share what you were trying to do when that happened?

I wish I had the transcript. I don't, and I'm afraid that the passage of time has muddied the interaction to the point of uselessness (when it comes to listing specifics).

Okay, so at this point it is strictly worse than just searching for and reading the very simple docs for the Ruby CSV parser, surely?

Because, as part of your verification, you will have to do that _anyway_.

And that may be where the discrepancy comes in. You feel fast because, whoa I created this whole scheduler in ten seconds! But the you also have to spend an hour code reviewing that scheduler, which, still it feels fast to have a good working scheduler in such a short time. But without AI, maybe it feels slow to find and integrate with some existing scheduling library, but in wall clock time it was the same.

The trick is that no one is actually carefully reviewing this stuff. Reviewing code is properly extremely hard. I'd say even harder than writing it from scratch. But there's no minimum amount of work you have to do. If you just do a quick skim over the result, no one will know you didn't carefully review every single detail. Then it gets merged to production full of mistakes.

To add to this:

If I as a reviewer don’t know if the author used AI, I can’t even assume a single human (typically the author) has even read any or major parts of the code. I could be the first person reviewing it.

Not that it’s a great assumption to make, but it’s also fair to take a PR and register that the author wrote it, understands it, and considers it ready for production. So much work, outside of tech as well, is built on trust at least in part.

I find this disrespectful by the author. I’m sure I’ve had colleagues at work that did this to me: throwing ai generated code at the reviewers with the mindset like "why should I look at it? That's what the reviewer does anyway".

I always passively call out the submitter on this stuff with comments like "Can you explain to me why you did this? Can you explain what this is expected to return" etc.

Usually gets them to sort out their behavior without directly making accusations that could be incorrect. If they really did write or strongly review the code, those questions are easy to answer.