In my experience 4o/Claude are really good at one shotting complicated but isolated components (eg streaming JSON parsers).

It does spectacular job with well trodden paths. Asked it to give me a map react control with points plotted and got something working in a jiffy.

I was trying to get it write robot framework code earlier and it was remarkably terrible. I would point out an obvious problem, it would replace the code with something even more spectacularly wrong.

When I pointed out the new error, it just gave me the exact same old code.

This happened again and again.

It was almost entirely useless.

Really showed how the sausage is made, this generation of AI is just regurgitation of patterns it stole from other people.

In my experience 4o is really good at ignoring user-provided corrections and insanely regurgitating the same code (and/or the same problems) over and over again.

ChatGPT 4 does much better with corrections, as does Claude. 4o is a pox.

That 4o is often times worse that GPT 4 has been widely ignored. :/

Chatgpt is great for writing regular expressions by the way.

Just because it spits out a RE doesn't mean the RE is what you wanted. For one thing, you'll need to be precise in your prompt.

You don't need to be precise. Just give it an example string and tell it what information you want to extract from it and it usually works. It is just way faster than doing it manually.

Claude/ChatGPT has become my man pages.

Yeah, in line with the old RE adage: I had a problem. I used AI. Now I have two problems.

Rhetorical question: Can you ever fully be sure you have the regex you wanted?

Writing regexes by hand is hard so there will always be some level of testing involved. But reading a regex and verifying it works is easier than writing one from scratch.

My overly snide point about regexes was that most of the time "verifying it works" is more like finding and fixing a few more edge cases on the asymptotic journey towards no more brokenness.

Yeah I agree with that! 100% test coverage seems impossible when every part of the regex is basically an if condition.

I've been using it to debug issues with config files and stuff. I just provide all the config files and error log to Chatgpt and it give a few possibilities which I fix or confirm is not an issue. If it still fails, I send the updated config files and error logs and get a new reply and repeat.