For example, image generators are watched for NSFW content by a separate AI critic.
It’s incredibly easy, you ask “did this answer solve your issue?” and add a max_tries.
> … how do you know how much training data to generate …?
You don’t, you keep doing it until the results improve to meet your goals, or they stop short and you switch tactics.
That’s why they have a fleet (parallelization) and why they outfitted the cars with sensors before self-driving was a thing (so they could simulate decision-making and have it corrected by driver action).
Their customers’ feedback absolutely trained their models.
The challenge is when AI has to interpret questions about stuff which can be expressed in syntactically similar ways with very different or precisely opposite meanings so it's very confidently (and plausibly) wrong about stuff like price changes and tax, event timings, refunds etc.
How does the model know when a human has to take over?
I think most extrapolations of current "AI" capabilities into future capabilities are fun and useful in some ways, but also doomed to fail. It's very easy to miss a tiny detail which may in practice be a fundamental problem.
> Use the actual usage data as training input.
Given that those bigger state-of-the-art models train on terabytes of data, how would you know how much training data to generate to sufficiently change the output?
My understanding of "AI" is that it's mostly about some very complex models which are capable of solving previously unsolvable problems. However, those problems are always extremely specific. Going the other way of thinking of problems or future possibilities first and then applying "AI" to it is likely to fail.