https://mobile.abc.net.au/news/2017-03-03/centrelink-debt-co...

College admissions? Resume screens? Mortgage applications?

We already have that at scale. There wouldn't be so much money in gaming those things if the AI wasn't doing such a poor job. You likely just happen to be someone that isn't on the wrong side of it.

This isn't true at all. When the process works well, those who it excludes often attempt to spend money to subvert it. There's no reason that interfering with the process should only happen when it's working poorly.

A simple example is the recent college admissions scandal. You often hear this scandal demonstrates that college admissions are just a way to let in rich people, but actually it demonstrates the opposite -- that the normal admissions process had (presumably correctly?) kept these people out and so they had to pay to subvert it.

Of course, you're not claiming that these things are just ways of separating the rich from the rest; just that they're messed up in some way or another. Which, if not very specific, seems broadly accurate. But the following claim isn't -- people on the losing end of a judgment are going to do their best to subvert that judgment whether it's correct or not, so one cannot infer much of anything about the judgment's correctness based on whether there's then an attempt at subverting it.

Garbage in, garbage out.

GIGO

If your physical bureaucratic entity delegates decision making to an AI, you have no hope of redress unless due process was mandated.

If you cut the bureaucratic middleman then you have to build an AI that can question itself and correct errors.

AI will never do that. So what you get instead is negligence because a computer makes a mathematical computation and the human element treats it as infallble.

I do not mean this to be about finite resources and our planet. I am referring to the way that organizations require more layers of bureaucracy as they grow. A rule of thumb is for every 10x in people, you have another layer of management. So what will this extra layer of management be for people, not just from a government perspective but also a capitalistic corporation perspective? How do Google, Facebook, Apple, Stripe, Amazon, and all of those other companies handle so many customers? How much do they use automation, how many of their rules are archaic, how hard is it to challenge the system if you think there was an error?

I am not convinced AI can solve many of the fundamental problems humanity are facing - and I think overpopulation might actually be the problem. Modern capitalism has conditioned us to believe "growth is good" but growth for the sake of growth is no different from cancer.

As an example I had to drop out of education (in the US) due to mental health issues. 6-7 years later, I'm still having problems getting back in as I wasn't able to get the problems documented in time. I also am unable to get any money as most of my classmates were despite my far above average performance pre-uni.

Which is why it's important to ask the question "should we" instead of just "can we".

Even more important is not dividing people into groups that we treat differently. Everyone should have the same education, work, credit and health opportunities. If we can free ourselves from the desire to segregate and separate people in the name of progress, whether or not an "AI" guesses wrong about you is irrelevant.

It comes down to ego... people have to accept that no one can predict someone's potential or value to the world with enough certainty to make restricting people's opportunities worth it.

Algorithms can be bad as well, but at least you can look into them to understand the reasons.

And that's just basic algorithms and regular human-written code.

Then you've got ML systems which are a whole 'nother level of inscrutable.