In this case, such a tool can be used to flag behaviour automatically (Facebook does that, for instance) to start a process, i.e. have a conversation. “Depression” does not have hard-set limits, and caring about someone who had a bad day is not problematic, talking about how to deal with rejection is appropriate. Forcing them to take mind-altering substance is not ideal, but I can’t imagine any licensed doctor doing that just because some patient’s score is high, even if they do not qualify otherwise; they receive a decade of training to teach them nuance. As someone who lives with a psychiatrist, I can confirm: no one is skeptical of classification any more than the people doing the rating.
Having measuring tools (imperfect as they may be initially) is what allows science to try opposable theories, and psychiatry needs this (and plenty more tools).
And perhaps the people being rated.
There are a lot of things to consider, here, those are exciting times for thinking.
But you can at least sue a doctor and get his license revoked. It is much harder to sue an algorithm.
Are you sure you have thought through all implications? I mean, getting locked up sounds pretty scary, but there are a lot of other scary stories that can happen if you really ban the whole concept of involuntary treatments.
What do you think doing A/B testing on cohorts of users is?
Advertisers are already doing this on a massive scale.
Facebook has shown an interest in manipulating depressive states of it's users.
There is behind machine learning both a phantasm - artificial intelligences that can guess what humans can't - and a reality - it's just statistical models that are never 100% accurate because, well, that's an attribute of statistics.
Those two elements combined and applied to behavior classification sounds like a scary thing, not unlike the kind of errors eugenics made, over trusting their science to apply it on social facts, totally discarding empathy and individual context.