How do you use it, as in, hey, write a doc about this, or do you iterate more like a conversation?

I do the second approach for coding with smallish steps and the output is fine

To be honest, I don't use it at all professionally except I guess transcription or search engine summaries to point me to actual documents. I mostly come across it when a client makes a contract or legal memo on his own and sends it to me to review. If I'm drafting a lease or a trust or whatever, I already have forms that cover everything they need, have been used thousands of times, and can be easily tweaked as needed cuz I half have them memorized at this point.

I can't imagine using them for a court filing. Those are either so short that they're trivial or they require painstaking research, precision, and often citations for every single sentence.

I’m against “vibe” anything important, but the fundamental flaw with this reasoning is that unknown unknowns exist.

I can’t cite “from scratch” for something outside of my knowledge but I side LLM training or assisted search.

It's not really any different in programming. Like if you have a well structured code and want to do a clear refactoring across it and you know what to expect, it can speed things up. But if it's generating any significant (and relatively complex) new code, you have to go through the whole thing manually again and then you find out you have to fix way to many things and get bogged down in different paths the AI didn't do correctly.

Of course, it's pretty much impossible to hear a dissenting point of view today and everyone is going crazy on these drugs. I might be hilariously wrong but I think this is the best time to start a software company.

Youre not wrong I believe.

I think its the perfect time to be contrarian - think about it. If youre wrong - So what? The world will have changed for everyone in the field. If you are right? You stand to be positioned to win big financially whilst everyone elses brain is rotting away.

Fact-checking and editing a mediocre piece of writing be way harder than writing from scratch. Proving that something isn’t true or can’t be substantiated is hard work, and so is arguing that a word choice is subtly inappropriate.

And making a ton of corrections to a document everyone was hoping was ready to go is never fun politically.

I have experienced this several times lately when writing software with claude/codex. Sometimes vetting and steering the agent takes longer than it would have taken me if done manually. Sure you can just decide not to vet the output and go into full vibecode, but agents tend to do a lot of dumb things (such as not deleting unused private methods or having temporary variables that are not needed).

In my experience the most effective work pattern for me is using agents to perform research and feedback on high level design, then I write the code manually, then I ask the agent to review the code for potential bugs/issues and fix those. The agents have a much easier time making small changes once the design is 90% there without going fully off the rails and generating slop.

I am working on writing skills to make the agent better but it is a bit painstaking. For example I had to write this inside of a skill because sometimes the agent would just stub out methods and leave TODOs: “always fully complete the requested task before finishing edits unless input is needed”.

This is the realization I had too. We had a manager update a policy at our org. He just shit it out through AI. It had tons of mistakes, people who read it had questions. Not only did it have mistakes it was causing people to do things in a way that added a manual step when an automatic process existed. Then the engineer VP commented on it asking the original author what its about who then had to bring it back up to the attention of the manager who made the first change.

It wasted many people's time, probably an order of magnitude of time wasted (and money) than if the initial person put a modicum of effort into making it right in the first place. Instead they hand it off to their life partner claude and just assume its good enough.

It's to the point where I am feeling insulted when I get ai slop like this from people. If I am expected to perform at a high level then I expect that at the very minimum the slop throwers will proof read their slop.

Ugh same. I take pride in delivering software that is bug free, performant, and to spec.

When our product managers just send us AI generated JIRA tickets that are extremely long but contain no actual details and tons of irrelevant or wrong content I get extremely frustrated and are seen as not a team player. At that point I’d rather not have the product managers present in the process.

You can also feed the document or source file to another frontier-level model, ideally two others, and tell it to vet it aggressively. The goal is to goad the models into erring on the side of false positive findings rather than potentially missing true positives.

I find that if Gemini Pro agrees with Claude Opus 4.8 and GPT 5.5 on something, it's almost certainly correct at a level where I wouldn't be likely to catch any errors myself.

Another attorney here. I understand your plight. But I can't believe law firms are sending out briefs and opinions without carefully checking all of the citations. I mean, even when Lexis or Westlaw identifies an (actual) case on point, you still have to check if the case has been overturned, whether it is truly on point, or if it can be distinuished from your case. So even if the cited case is not a halucination, someone would still have to read and analyze the cited case in the context of the present case.

>> But I can't believe law firms are sending out briefs and opinions without carefully checking all of the citations.

Update your priors: https://www.damiencharlotin.com/hallucinations/

Be afraid, be very afraid:

"AI Hallucination Cases" - https://www.damiencharlotin.com/hallucinations/

>As an attorney, I feel like vetting AI output takes longer than just doing it from scratch, let alone versus just using a traditional form.

This is my issue with AI.

In the type of work i do the work needs to be precise down to the context of how individual words are used.

Having AI pump out 20 pages of content but then me having to go through the 20 pages word by word, cross checking references and prior statements is going to take a long time.

Not to mention I didn’t write it, so my brain doesn’t already know what’s been written so it takes several passes to confirm its complete and that it all fits together.

I find it easy to just write it myself use AI for the more menial tasks like logic check, completeness checks, etc.

One task AI was very useful in was where we wanted to understand the gaps in a submission relative to a process requirement document. We didn’t care if the output was 100% complete or perfect, we just wanted a few examples.

Being able to input a couple 300 pages documents and have AI spit out a dozen examples in 30 seconds was a huge time saver.

But that was a non-critical task.