I don't think the two hands can properly "validate" algorithms without communicating. The algorithm designer can maximize AUC, but what if one <group>'s class is 95% label A; the designer always predicts A for <group>. How bad is ALWAYS missing 5% for label B? If you can put a price on it, then the developer can build it into the algorithm. But if the price is difficult to accurately estimate, or non-monetary qualities are desirable, it may be hard to build them into the classifier ahead of time. On the other hand if the cost of perfect <hard to quantify criterion> reduces AUC significantly, algorithm designers need to communicate that...
Humans produce the data that the ML algorithms train on, so without whitening the data somehow (and there's another contentious issue) we should expect the result to be biased as well.
How do you guys think this is going to be handled? It seems very complicated, because how are we even going to know which algorithm is at fault, or even IF any can be conclusively determined to be at fault?
Furthermore, is he talking about Bob the implementer or Bob the approver? In many organizations the people actually writing the code are not the same people who determined the specs and asked for it. Bob the implementer may have been given a limited spec that is followed, unknowing there is potential that an outside influence can manipulate the results. At that point who holds the responsibility?
Maybe it's just me, but recently I have noticed increased discussion regarding ethics in tech. It seems like there is growing awareness that all the focus on "how" to do something has eclipsed the fundamental question of "if" we should do it. This isn't to say we should just shy away from learning algorithms and complex systems, but maybe we should be put more of an effort to discuss the outcomes and how we agree to handle them before charging full steam ahead into uncharted territory.