A couple of reasons. Essentially you are asking “what is wrong with a perfect system” but the problem is no system is perfect, and we spend most of our time dealing with the imperfections.

In my previous jobs the consequences of me making a mistake were that a page didn’t load or some content was unavailable or in the worst case deleted. But a mistake in a loan decisioning system is likely to be 1) much harder to detect and therefore 2) much more likely to persist but most importantly 3) vastly more consequential.

Second, there is no way to determine from first principles whether someone will repay a loan. So you will need to use historical data. Loan data is very messy and biased due to cascading historical legal and quasi-legal policies. You may believe you can erase one feature from the training data (e.g. race) but if your model can infer race from other factors (which is likely because other features are not randomly distributed across populations) then you have the same problem you started with.

To take a concrete example, there was a man who was wrongfully accused of fraud. Then after he won his lawsuit, he was wrongfully accused trying to cash his damages check.

https://abcnews.go.com/amp/US/black-man-sues-detroit-bank-al...

Some amount of racism, classism, and negligence went into that decision. That’s the bad part. But the good part is we could see what was happening. If instead the process was automated, it would be a lot harder to prove the same problem.

There was a comment in HN a couple of days ago where someone said they had a hard time getting an apartment for years until they finally asked what was going on, then had a hard time fixing someone else's mistake. The computer was erroneously telling all of the leasing offices that the HN user owed someone thousands of dollars.

Garbage in, garbage out.

>> diffuse responsibility

My first problem is somewhat dogmatic, which is just that it feels bad to interact with a robot who has power over you. At scale I fear widespread psychological effects, feelings of lost agency, impotence. Haven't you ever tried to navigate an automated phone system and had the robot hang up on you? With no way to contact a human? It can be infuriating even if it's a simple customer support request, let alone money being on the line. Somebody once shot up the YouTube offices because he made income on his channel which was unceremoniously closed without recourse. I'm not making an excuse for the shooter, but pointing to him as an example of the psychological consequences of automated "support"

My second problem is more practical: talent pool. When a task is manual, people who retire have to teach the newbies how it works. Experience is transferred and at all times there are people who understand the system and how to fix it when it breaks. Once a task is automated, there is no need to train the next generation of talent, a computer can do it better than a human right? Until a couple of workforce turnovers down the line and the system goes on the fritz -- no one knows enough about the system to even recognize that it's failing at its task.