I put agency in quotes because it's really a convincing illusion of agency that we're going for. In the end, I agree with those making the point that even we don't have free will.

Ultimately it just has to be able to convince humans that "wow, there's an actual thinking and learning 'being' in there."