In my specific specialty - AWS - I asked it a lot of questions like I would at an interview and it answered the questions well enough that I would have passed a candidate who gave me the same answers.

I didn’t just ask definitions. I asked your standard compare and contrast, why would you use, is X and appropriate use of Y, etc.

It can also write Python code, modify existing code based on plain English sentences and do passable code reviews when I asked it “how would you improve this code” when I gave it a 200 line script I wrote.

Then I asked it to make the changes it suggested like unhardcode some values, add exception handling, test for existence of a property it was trying to access.

That's awesome, it does seem to be quite good at writing code. But in that case, you can test for it's weaknesses by asking it anything recent. The models are only trained on data through X date. Ask it something after X and it wont have the data to answer you correctly. Like about some new framework or feature.

Hell I work at AWS in Professional Services and you could ask me anything recent to test my weaknesses. I can’t keep up with all of the announcements.

Google can though :)

The context here was comparing ChatGPT and Google.

I don't think people are out there producing fake theories for how to use AWS.

But if you try something medical, or god forbid political, it's dataset is not pure, it constains falsehood perpetuates by laymen, urban legends and outright fraud.

Can you give an example? In my limited experience, it refuses to answer anything political.