But with code, it’s easy enough to prove correctness just by running it.
That being said, the one bug I find consistently is that with the AWS APIs, all list type methods pagínate and you have to account for that. Python/boto3 have built in paginators.
This an insidious bug because things will work correctly in a dev account with only few resources. But will fail in hard to debug ways in production.
What’s even worse is that ChatGPT “knows” the pattern and will correct itself once you say something like
“This won’t work with more than 50 roles/ec2 instances, etc”
I may not know always know the correct API or CloudFormation/CDK/Terraform syntax. But if it gets it wrong, I can read the docs and correct it.
Providing sources doesn’t usually help. ChatGPT consistently makes up sources.
Certainly with code the proof is in the pudding, but most recently my problem was "I need to create AWS monitors in Datadog to alert when a region is down." ChatGPT was hopeless but bard was able to point me to the exact doc explaining how to set it up.
It seems to give answers that are 100% incorrect and when told so says "of course you're right, here is the right answer". The only stuff I'd use it for is if I already know exactly how to write the script and I'm just using it to type it for me quickly because I don't remember if aws_instance is the correct spelling or aws-instance in terraform...