The person operating the computer or building the AI could be held accountable though.

You could get a bystander effect kind of dilution of responsibility, even if unintentionally. Everyone points to someone else which is a system of incentives that enables bad things to happen. The buck always stops with the leadership, but you still want well-defined blame at the decision maker level that's transparent to everyone while decisions are being made.

Yes, and that's not happening right now, and it's a Big Fucking Problem. I am pretty sure this will someday be a prime case study in AI ethics courses. A Waco type of "how could this happen" moment.