Jokes aside, we are entering a world of AI-decision hell. If it is any similar to the algo-decision hell from the past decade, humans are fucked.

The racism mention in the article is notable - this isn't prima facie racism (and certainly affects more than minorities), but it is derivative racism since "people who can actually be heard to go through the system in the right way" is race-skewed.

If they can do it, so can we.

Just like offshoring manufacturing and service centres to places that pay the lowest wages (even though us proles aren't allowed to buy entertainment media from those same places, but that's a separate topic)

Race you to the bottom!

In the case of Google and Meta though don't confuse Customer with User. User service is not a thing.

Clearly this thread -is- about Customer (not User) Service, so I'm not suggesting that banks are the same.

Financial systems have a bot / scam / fraud problem. As others have pointed out, that is the cause of most of the human pain in that space.

Interpreting such results is a job. Disputing such results in a court, or with an arbiter, would be really hard, unless you're a practitioner of the pattern-recognition craft yourself.

Although I do think that the model weights and parameters should in some way be subpoenable so that the model can be 'compelled to testify' if someone has enough legal, technical and fiscal resources to sue the company.

Some of best examples are the juvenile and and family court systems in a state like Ohio. Not only do they not even TRY to do what is fair or right, they have one simple mandate, pay for their own existence. As soon as the system set up to punish children is made to profit off those same children you have a sick and broken system but thats where we are today. So, VERY little chance we are going to convince a company to make less money for the social good when even out children's courts are run like a business.

Standardized testing and remote learning monitoring, automated sentencing software, and any number of social programs are examples, if you were wondering.

So, part of the problem is the signalling that the tolerating of these practices when they happen to those less able to defend themselves presents to companies. "Go ahead, do it, you'll get away with it." I tend to think: because the people with the power to stop it benefit from it.

Have you ever accompanied a poor or disabled person to a county assistance office to see what the forms or process entails?

Do you know how SNAP or WIC works?

I never said "all".

Some years back, and acquaintance of mine detailed how he gamed the system. He'd work until he met the minimum number of employed weeks to qualify for unemployment, then he'd do a lousy job so he'd get laid off. He'd then go on unemployment. Being unemployed required him to look for a job, so he'd look for a job, and flunk the job interview. When the unemployment compensation ran out, then he'd get a job again. Over and over and over.

BTW, when he was "unemployed", he'd take off-the-books jobs.

A businessman acquaintance told me how he'd found some well qualified people he wanted to hire. They said they'd take the job, but only after their unemployment compensation ran out.

The Seattle Times ran an article a couple decades ago (wish I'd saved it) where they went around interviewing people on welfare. They'd ask "what would you do if the welfare was cut off?" The answer was "get a job." They interviewed people in a car with fishing gear. "Where are you going?" "Fishing" "Don't you have jobs?" "No, we get welfare." "Wouldn't you rather have a job?" "No, we'd rather go fishing."

Sure, there are people who should be on welfare. But there are a lot of people gaming it.

Every government program designed to hand out money if specific requirements are met results in companies and individuals springing up to game it.

70% (approximately) of people on welfare already have a job. That's strange isn't it? Not going fishing. Not living the life working under the table. No, doing the hard jobs you expect to be done for next to nothing, for your comfort and convenience. They clean your hotel rooms, wait on you at restaurants and check you out at your local Walmart that destroyed all the other work in their town, and then paid them so little they had to apply for help to feed their children. These vital jobs that people expected to be done even in the middle of a pandemic, well they don't pay enough to live off of anymore. Because the wealth only moves in one direction and has for decades.

Welfare the majority of the time is a hand up not a hand out, 50% of recipients are able to get off of it inside of a year, 70% inside of 2 years and if we should have a few people who game the system, oh well. It's a necessary evil to make sure women and children are not on the streets, to ensure that once you are down you are not out, because we are the most prosperous nation on earth and we can EASILY afford it, and part of the social contact entailed in being an American is you care for your fellow Americans. Unless you are someone who sits around ruminating about what an injustice it is that a portion of your income is used to keep children from being hungry and how very unfair that is to you.

I pay more than what I am asked in taxes, I contribute to charities at a rate that most people would be surprised by, and I can tell you this. Every charity in our country is a failure of our government. Every church that proselytizes via helping the poor does so because the system we built is not working for it's constituents, and myopic people want to spend a bunch of time fighting about who gets what of the very, very thin slice of the pie, instead of actually doing the hard work of reforming the bakery. If there is any truth to your anecdotes at all consider this; it might be more telling of you and the moral fibre of those you surround yourself with than people at large.

What happened in 1968? Johnson's Great Society program.

https://www.eia.gov/dnav/pet/hist/LeafHandler.ashx?n=PET&s=M...

It’s much lower after adjusting for social benefits.

Right, and it stopped going down.

If you'd like to sanity-check your recollections, try to figure out what the people in question would be eligible for, and try to figure out what it would actually cost to live in Seattle.

Believe whatever you need to. But consider this:

People are fundamentally selfish. Building a system that assumes general altruistic behavior is not going to work very well. Assuming people will not take advantage of rules for their personal benefit is naive.

Consider politicians. Have you ever noticed that people tend to go into Congress as middle class people and come out wealthy? How do you think that happens? Altruism?

How many people offered student loan forgiveness will say "no thanks, I borrowed the money, I'll pay it back!"?

The people I mentioned were not breaking the law. But they were taking advantage of it.

If you're one of the few people in the world doing well enough to be paying into a system rather than relying on one, thank your lucky stars for how blessed you are. Ultimately, nobody on their deathbed contemplating their life cares about who their taxes were spent on.

If you simply must be mad at someone getting government handouts without need, there's a big list of companies who use accounting tricks to pay no taxes and even get subsidies from the government. You're probably better off being mad at them than some guy who's figured out how to go fishing while collecting unemployment.

I remember when the unit at the company I worked for decided to put a big graph of the bug count on the wall, and announced incentives for retiring bug reports. Within a week, the entire unit had devolved into knock-down drag out fights over who had solved a bug, if a bug was really a bug, if a bug was actually 3 bugs and the person should get 3 credits for solving it, people filing fake bug reports and then "fixing" them, etc. A couple more weeks of that and the graph was taken down and the incentive program terminated. These were intelligent, well-educated people, going berserk.

There was another case where the company got a big contract from IBM, and hired some mercenary contractors to fulfill it. Part of the deal was each mercenary got a $10,000 bonus (worth about $30,000 today) for getting it done on time. I'm sure you know what happened. It shipped the day before the deadline.

I asked one of them, "did the bonus get you to finish on time?" The response, "absolutely not, it had nothing to do with it. We're professionals!"

"For every 10,000 households participating in the Supplemental Nutrition Assistance Program (SNAP), about 14 contained a recipient who was investigated and determined to have committed fraud (via a 2018 report by the Congressional Research Service). Within SNAP, for every $10,000 paid in benefits, about $11 is determined by state agencies to have been overpaid due to recipient fraud.

To put this into perspective, the IRS estimates that for every $6 owed in federal taxes, $1 is not paid because of tax evasion or fraud."

Well, except the off-the-books job.

Fraud as defined by these authors is very rare. They are not talking about what the average person is when they talk about common welfare fraud. These studies mean outright fraud, not someone able body choosing to game the system instead of working.

If you changed the definition to match the collequal understanding of fraud your results would not be a number literally everyone who lives in neighborhoods where benefit recipients are the majority laughs at when told.

It's like saying the PPP program had very little outright fraud. True. But in my mind, you committed fraud if you took the money and didn't actually end up truly needing it for it's intended purpose. Most share my definition. Under that definition I'd posit the vast majority of PPP recipients participatated in fraud, but it will never show up in the studies or numbers.

Anyone who has spent any amount of time actually living within these communities understands the situation with benefits looks pretty much like the situation with PPP writ large.

That way someone who used aid and becomes wealthy would pay into it in the future. Right now our systems more or less just keep you where you are.

There is not a single program in the US, by careful design, that can give “too much”. Yet we still spend so much money on the bureaucracy trying to disqualify anyone from their own programs.

Imagine how much we’d save, socially (less homelessness! no death for lack of healthcare! a more educated and informed country!) but also maybe financially, if we actually _gave our people what they need_.

We don't need to imagine how that would work out. There are plenty of examples.

It's probably the most tried and failed system in history. There are even examples that predate Marx - the Jamestown and Plymouth settlements.

It's perfectly legal in the US to set up a commune. The people who want it are free to give it a go.

The money spent on fighting individuals could be offered as a top-up to the working poor which would other programs to raise levels.

1. Opt out of these things as much as you can. Often times you can't opt out of the underlying actual thing (school, banking, etc) but you can opt out of the terrible new thing. You don't need to explain to people a deep "why". Instead give the people polite, reasonable (and generally true) excuses. "Oh, it doesn't work on my computer". "Oh, I don't understand how to use it." "I don't really like technology."

2. Help other people, particularly children, the elderly, the vulnerable, etc, in opting out of these things and bypassing them when needed. Help them fill out the paperwork, call the right people, cause the right fuss. Frequently, you'll find by helping one person they'll go on to help other people do the same thing.

3. Tell other people, in appropriate times and circumstances, that you don't care for these things and opt out of them. Don't be weirdly obnoxious about it, but when someone gripes about it or complains about it not working for them.. just agree with them. Most people don't like these things. These things don't generally work (for anyone). They suck. Let other people know you agree with them that these things suck.

There are cases where this doesn't work - automated account closures, automated sentancing, rent shenanigans. You can of course do the usual things of bugging your congress critters (and you should) but you should also use the full range of pressure available to you. Don't do business with shitty companies - and when they inevitably try to sell to you, tell them why you don't do business with them. When a politician supports automated sentancing tools, explain to them why you aren't voting for them.

Surprisingly a little bit of pressure goes quite far.

how about we stop at the actual source and by not having the government launder the task of law enforcement to third parties? The article makes it pretty clear that banks aren't exiting customers for fun. They're doing it because they have to, on pain of facing regulatory action.

This is already done for financial statements.

If you put out software that is actively causing harm, you should know your fingerprints are on it

Employees like devs would be protected by professional insurance. That would cause bad management to shape up, or otherwise lose inaurance coverage.

Everyone has agency. “Just following orders” is not an excuse.

If someone asks you to implement something that crosses an ethical line for you, you ALWAYS have the agency to refuse. It’s hard, but it’s critical that we never give away our own agency and control of our own actions.

You may not be able to stop an implementation from going forward, but you ALWAYS have the ability to not contribute to such an implementation.

depends on your safety barrier. In some very high profile cases, depends on your life. I commend people like Snowden but I certainly wouldn't choose to spend my entire life running from the US government in an attempt to be ethical. I have too many selfish personal goals to try and attempt to sway the masses like that.

You wouldn't have to flee to Russia for refusing to ship spyware in a mobile app, for example.

It isn't an automated decision-making problem. Every instance involves humans in the loop making the final decision. And that article makes clear -- via spokesperson statements -- that it is auditable and the banks know exactly why accounts were terminated.

I agree but I wanted to push back on the idea that this was somehow due to automated decisions being made. That isn't really the problem here. The same exact thing would happen if the government mandated "no computers are allowed to be involved". The problem is opaque decision making, no clear escalation path, a disregard for the short term pains, and so on.

The fundamental problem isn't how the decision gets made. Even if a human made the decision you'd have the same issues of the front line consumer support saying "well the other department made their decision, I don't know why and I don't have the authority to change it and they don't accept calls".

By talking about automated decisions, the OP is making the conversation about the wrong problem.

Now, I do not know how bad fintech scene is, but I do know that they generally suck when it comes to actual BSA compliance. Then again, I do not really consider fintech banks. Neither does treasury for that matter[1].

Still, this is not THE problem even if you raised a valid and relevant concern here. The issue is and always has been BSA legislation that put this requirement on banks without specifics. More than that, the core issue is that the stupid population that was so scared post-9/11 that it let this thing pass through with minimal whimper.

Clearly, it got bad enough that it got some important people and Treasury had to issue a guidance[2]. No one cared for 2 decades.

I am sorry for the tone. I am glad this is getting a wider audience, but I do have a bee in my bonnet that it took this long.

[1]https://home.treasury.gov/news/press-releases/jy1105 [2]https://home.treasury.gov/news/press-releases/jy1438

As with any classification system though, 100% accuracy isn't going to happen. But there's always some customer service rep that can look at the details of the account, and see why in the world the system said what it did. But a detailed explanation of why we thought something was fraudulent could (and sometimes would!) just lead to another fun reddit post where someone describes how to hide the fraud a little better.

For any given system like this, how much harm is actually being done, vs how much is being prevented (as fraud just leads to raising prices to cover for it: financial companies are not charities)? I've read way too many CSR conversations where a blatant fraudster with world-class chutzpah would claim that we were destroying their family for no reason, when the data was damning. But this doesn't mean that everyone who isn't a fraudster really reaches out to the CSRs, and has the energy to prove there was no fraud. The actual levels of damage are just hard to measure.

We should have sensible, mandatory, available customer service access, which costs just enough to access to not be hammered by bots, but that is completely refunded in case of error. But what is really causing this is that many companies have lowered the barrier of interaction so much that we are letting a lot of fraud through the door. Remember how getting a merchant account in a real bank is a multi-day affair? How getting hired to become a delivery driver needed an interview, with a real person, and a manager checking between deliveries? The price of not having to interact with a human to sign in is fraud detection that isn't a boss you interact with every day, makes sure you are working, and is paid from the work you do. Companies with billions of customers and probably hundreds of millions of suppliers aren't exactly workable without automating a lot of those intermediate jobs away.

Maybe we made the wrong call across the board, and lower-productivity, but far higher trust commerce is the way to go... but a lot of that commerce is losing in the market, right now. So if we like it, we have to be willing to pay extra for it.

Plenty of people in the comments here want to enforce that approach via laws and regulation..

> So if we like it, we have to be willing to pay extra for it.

.. and I wonder if they are taking this into account.

That's what brands and reputation are for. And it works: brands often do command a premium.

(Mainland) China has regulations for milk. Including banning of chalk. Alas, those regulations aren't enforced with much teeth in practice, so consumers rely on reputation and brands. Specifically, they buy milks from oversees, like Australia, because of their superior reputation.

A big part of that reputation is that Australian companies won't be protected by the Chinese government when they screw up. So it's harder for them to hide blemishes on their reputation (at least to hide them from Chinese customers).

I worked for a company doing this sort of fraud detection. Knowing features and weights really would make it easier for fraudsters.

If I were in charge, it'd be too bad for your company, and they'd have to give a detailed explanation every time even if that would be the result, because the alternative is way worse.

Because it is a legislative problem of not ensuring sufficient rights, such as the inalienable right to have access to an electronic money account with which you can receive, send, and keep money.

    > human oversight policies provide a false sense of security in
    > adopting algorithms and enable vendors and agencies to shirk
    > accountability for algorithmic harms.

    > propose a shift from human oversight to institutional oversight as
    > the central mechanism for regulating government algorithms.
    > First, agencies must justify that it is appropriate to incorporate
    > an algorithm into decision-making and that any proposed forms of
    > human oversight are supported by empirical evidence. Second, these
    > justifications must receive democratic review and approval before
    > the agency can adopt the algorithm.