It's a little bit more complicated than that, it was "the rules said so" and then they just used humans to check if people used the rules. The government at the time was very hard on fraud after a couple of high-profile fraud cases by immigrants. Mostly Bulgarians falsely claiming welfare

Anyway, several ministers and other high officials pressured the welfare department to be hard on fraud, which caused the imposition of harsh fines for minor filing errors. The department also got into a bit of a witch hunt were anyone being investigated for fraud was almost always presumed to be guilty. This, combined with the fact that most of the welfare recipients were not very educated, didn't speak Dutch all that well and/or were under a lot of stress due to being generally economically disadvantaged, led to some big fines levied against people who didn't objectively do all that much wrong. Several of them lost their homes or got deep into debt due to the fines and legal costs.

Simply claiming it was "the algorithm said so" skips over all the history about why the rules were written the way they were, and the human errors made along the way.

Yes, there definitely was and it wasn't so much driven by the tech itself as it was by the need to be seen to be doing something against perceived fraud by minorities which led to mass profiling and huge fall-out (and too little, in my opinion). It was effectively racism under a thin veneer of plausible deniability provided by technology.

Tech by itself can be bad enough, tech wielded to achieve nebulous goals without any consideration for the human side of the effects is outright dangerous.

> by the need to be seen to be doing something against perceived fraud by minorities which led to mass profiling and huge fall-out

It was not only perceived fraud by minorities:

https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scand...

https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scand...

Most practitioners can tell you that naive anti-crime initiatives often end up targeting minorities. In many Western countries, minorities are significantly over-represented in crime and jail statistics (compared to the general population). Is that ipso facto racism or racial profiling?

This is exactly the kind of thinking that caused this whole thing to spiral out of control.

The little bit of fraud that was uncovered did not justify declaring a very large fraction of the Dutch population as fraudulent. I'm kind of surprised that your takeaway is exactly of the kind that has been proven to be utterly wrong and counterproductive during this whole affair. Yes, what you are advocating for is racism and racial profiling.

Indeed, though that wasn't AI, that was direct discrimination and racism hidden behind a thin layer of policy.

https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scand...

Indeed! https://en.wikipedia.org/wiki/Dutch_childcare_benefits_scand...