News for hackers. Stuff that matters.
Re:Stop Explaining Black Box Models for High Stakes Decisions by soVeryTired on Friday November 01, 2019 @09:06AM (#21418270)
> The higher the stakes of the decision, the more incentive to approach it like a rational Bayesian agent who would use whatever tool has the best risk / reward tradeoff, totally regardless of explainability

These sorts of problems are mostly difficult precisely because we don't know the risk/reward tradeoff.

For example, in the prison system you need to evaluate the risk of recidivism. How exactly do you evaluate the tradeoff between releasing someone who is potentially violent vs. keeping someone potentially reformed behind bars? You'd need to weigh the dollar value of keeping them imprisoned and the personal harm to them and the rest of the prison population when they're imprisoned against the potential harm to anyone else if they're released... plus many other factors that I'm sure I've overlooked.

Anyone who tries to bake those tradeoffs into a "rational Bayesian agent model" will fail.

You're trying to describe the world in terms of game theory, but we live in a world where we usually don't know the risks, or the payoffs, or the rules of the game.

Re:Stop Explaining Black Box Models for High Stakes Decisions by mlthoughts2018 on Saturday November 02, 2019 @02:34PM (#21429223)
> “These sorts of problems are mostly difficult precisely because we don't know the risk/reward tradeoff.”

If you don’t know this trade-off, then you don’t have a decision problem.

Basically what you’re saying here doesn’t add up at all. Issues where an ML model is used but people want more “explainability” are places where we absolutely do know the risk / reward characteristics, and we’re specifically trying to measure more of the relationship between the internals of the decision process and those components of the risk / reward characteristics (for purposes of political arguments over who controls the process, dressed up as if they are scientific inquiry instead of politics).

If you did not even know what objective you are pursuing at all, you would not be using an ML model, and likely could not even identify what type of decision process you even need yet.

Re:Stop Explaining Black Box Models for High Stakes Decisions by soVeryTired on Monday November 04, 2019 @10:59AM (#21442808)
> If you don’t know this trade-off, then you don’t have a decision problem

Let me clarify what I mean.

In the cases I've worked with professionally (finance, autonomous driving), no-one gives you a reward function. You have a rough idea of the outcome you want, and you fiddle around with reward functions until your algirithm kinda-sorta delivers the outcomes you knew you wanted in the first place.

In this sense, the OP gets it precisely backwards. OP wants some sort of hyper-rational Bayesian model that optimises outcomes based on the one true loss function.

There is no correct reward function, and objective functions in general tend to be far too simple to capture the nuance one encounters in the real world.

> If you did not even know what objective you are pursuing at all, you would not be using an ML model...

It's not that you don't know at all, but you should be very clear that your objective is a fudge that probably doesn't capture what you really want. Maths is clean: the real world is messy.