But they can be undone with pencil and paper too. The footgun of automation here can only be undone with either a really good patch (git commit -m 'finally finally works for real this time') or a lot of pencil and paper work that's slower than the processes that caused the problem.

I'll note that this isn't the first time that people have said "well its the algorithm" when they were responsible. The example that springs to mind is bail risk assessments. You're very correct in that there are people making real decisions that are very cruel here. The machines give them something to hide behind.

Yes. "It's just the algorithm" is the "it's just procedure" of the 21st century.

Except software allows a scale and efficiency that is impossible with pencil and paper while also creating an ideal scapegoat. Software is being used to avoid accountability at a scale much greater than what was possible with manual process.

The old saw about computers is that they're designed to allow humans to make millions of mistakes, very quickly and very accurately.

> Software is being used to avoid accountability

Right, in the same way knives are used to rob people.

This is not a new problem: an organization strategically builds an unmanageable bureacracy and then profits off the issue while claiming incompetence.

Computers just make said bureaucracy cheaper to operate.

Isn't this the basis of the book "IBM and the Holocaust"[1] where the author lays out how IBM's technology helped facilitate the wide scale of Nazi genocide?

[1] https://en.wikipedia.org/wiki/IBM_and_the_Holocaust

How is this a government problem? People frequently lose access to social network accounts and email because of broken algorithms. Google can blacklist a business and send it broke. Insurance companies, credit bureaus and banks can make a wrong decision and deny credit.

People decide to fix problems in software or anything else for that matter. If they don't get fixed it's usually a matter of somebody (or bodies) decided it was not a priority.

Corporations have similar issues. Just look at the biased image recognition technology that FAANG release