If you are employing someone to build software with AI, then interview that way.

I think what I'm looking for these days is people with a technical mindset and also can think strategically. So its not Yolo prompting, but creating an engineered environment for creating software for a particular purpose.

So if you want a good AI Engineer you want to test that can address the following things (not exactly, or everything, but they should be covering a lot of these things in some form) :-

  - Define goals
  - Distributing/Managing work in a team
  - Establish guardrails
  - Build feedback loops
  - Use multiple agents, spanning out AI then recombining
  - Setting up Verification
  - Evaluating Engineering (Security, Architecture, Code, Performance, Deployment, Monitoring )
  - Set it up that the whole process improves itself 
  - Understanding Domains
  - Mitigations for an AI outage.