A tool designed to find people guilty is biased to find people guilty.

As far as I know it is fairly easy to take a generic dna sequencer meant for healtcare diagnostics, and repurpose it for STR analysis. The only major difference between the healthcare versions and the forensic versions is the software i/o.

> A tool designed to find people guilty is biased to find people guilty.

I don't see those particular issues make it biased, just inaccurate - it could go either way.

And yet somehow whenever you take a closer look at mislabeled product prices, the average is always in favor of the store. And that's far from the only industry.

Complex tools are the product of many thousands of individual decisions taken by humans, humans aware of who's the paying client.

> And yet somehow whenever you take a closer look at mislabeled product prices, the average is always in favor of the store.

This could just as easily be selection bias: the errors in favour of the customer are less likely to get reported by customers.

> And yet somehow whenever you take a closer look at mislabeled product prices, the average is always in favor of the store.

What is this based on?

I had a 'fun' experience along these lines with health insurance and medical bills a couple years ago. I can confirm that in our case at least, /every/ error we found was not in our favor, and took usually about an hour on the phone to get fixed.

The somewhat-less-malicious interpretation is that the companies have a strong incentive to detect + fix errors that cost them money. Meanwhile, consumers are a) non-centralized, uncoordinated, and often unaware of errors, and b) have no way to fix systemic issues that impact them. And the companies therefore have no /real/ incentive to fix systemic problems. It is literally more profitable to fix the bills of the few people who complain, as they still make money on the remainder who don't notice the errors in the first place.

(on edit; exactly what the other comment one subthread over said. :P )

decades of data?

I still believe Hanlons razor applies. I've seen products that have serious performance affecting bugs caused by similar mistakes.

I still think—even when applying Hanlon's razor—there's an imbalance in incentives that leads to a weight in favor of the interests of the party paying for the test.

Take the store pricing example. Suppose the store's pricing & labeling process produce an equal number of bugs at checkout in favor of the store and in opposition to the store.

The store is heavily incentivized to detect the errors that are opposed to them. They are much less likely to detect the errors in their favor. Consider the manager that looks at the cash at the end of the day and notices they are $500 short. They likely dig hard to find the root cause of the issue, detect the pricing disparity and correct it. Now consider the manager that is $500 over at the end of the day. They are much more likely to say: "that's weird", shrug their shoulders and move on.

The same applies to forensic tools. Even if they originally produced bugs in both directions, their own internal QA and the market of police officers are likely to work hard to detect bugs that make them less likely to allow them to make an arrest.

The net result is that the tools end up with a bias in one direction, even if the original developers made an equal number of mistakes in both directions.

Most store managers get as grumpy about overages as undercounts. They mean that some customer got shortchanged. For $500, it probably means a lot of customers got shortchanged, or something even worse is going on. That makes customers grumpy, and it affects your future.

There are plenty of lazy managers who would sweep it under the rug once. But if it happens more than once, it can become their job on the line. They start looking for who's counting wrong. And if they can't figure that out, they get really worried.

I have no idea about police officers and prosecutors. But store managers care about accuracy of counts, not just profits.

When running an experiment and following poor practices (i.e. p-hacking), results that fit the hypothesis will be accepted more readily and negative results will be debugged or re-ran more often.

i.e. The initial error may be randomly distributed. But the follow-up on the error will have a lot of bias.

This is also similar to how Toyota hilled people with control software that would cause car to accelerate randomly. The software audit team concluded that they could not find the bug, but the code was totally unreadable and terrible. They settled.

https://www.nytimes.com/2013/10/26/business/toyota-agrees-to....

Also let's remember that a company in UK was selling fake bomb detectors to Israeli and other militaries, and it took them more than 10 years to notice!

https://www.bbc.co.uk/news/uk-29459896

There needs to be proper scrutiny into these things, I could start some random 'deep learning to find criminals' company tomorrow, and have less regulation than a car mechanic

Every attempt to replicate this has failed, and other vehicle manufacturers have just as many 'sudden acceleration events'.

https://en.wikipedia.org/wiki/Sudden_unintended_acceleration

Occam's razer points to people just hitting the wrong pedal or people's floor mats getting stuck.

The reason Toyota ate it in the press for this was competitive.

> The reason Toyota ate it in the press for this was competitive.

can you explain what you mean? I don't understand this sentence.

I think they mean this: Toyota “ate it” (an idiom meaning to take losses) in the press by having bad stories written about them. These stories were written for a reason. This reason was not that Toyota deserving those stories, but that Toyota’s competition encouraged those stories.

From what I've seen of the Toyota case, the thing that Toyota actually got slammed on had nothing to do with control software, but was about the mechanical design of the pedals and the floor mats.

I wouldn't be surprised if "incorrectly averaging" and similar are very common software errors.

The reasons are manifold, including:

- Normalized values need to be averaged differently the absolute values.

- Floating point has limited precision, even just correctly summing/multiplying numbers need special care if you care about correctness. Results can, in the worst case, be of by a massive amount.

Often you don't need to care about it so it's not uncommon for especially junior programmers to be not so aware about it.

I mean in the last 3 years of working as a professional software engineer/developer I didn't need any of this at all, but once I do I know what to look out for.

Draeger requires calibration prior each use for temperature and humidity. It is easy to get it thrown out as evidence.

When the officer shows up and swears under oath that he "calibrated it that morning" it's not so easy. :shrug:

Logs of calibration procedures along with calibration results should be stored on the device and auditable along with test results.

Temps vary throughout the day.

A rural small town judge probably doesn't have the best insight into that, just that Corporal Bubba said he calibrated it so it must've been right.

I know, this is exceedingly cynical.

Your "Corporal Bubba" isn't just cynical, it leans heavily upon a harmful stereotype that folks in small towns are uneducated simpletons. Stop the polarization, please

Why wouldn't they be? I went to university with people from all over Russia. They don't tend to return to their home towns after getting educated. Does this work differently in America?

> Does this work differently in America?

No, not really.

Not all poor people are stupid.

Is hatefulness an end in itself for you?

Maybe. Thanks, I could probably use some self-reflection.

(The anti-stress effect of covid vaccination seems to be much more immediate than I expected. This is the second time today I find myself saying things highly unusual for people on the Internet in general and my yesterday's self in particular, and the first time was literally a couple of minutes after the procedure.)

I'm glad that you're feeling better :)

I'd love it if more cops had college degrees in the US, but they don't. Ignoring the problem doesn't make it go away.

Sadly, when you consider the available evidence, Corporal Bubba isn’t too far from the truth. I say that as someone whose Dad is a retired police officer and who spent most of his youth in a very small town with sub-1200 people.

Rural areas are subject to brain drain where the best and the brightest disproportionately leave for more urban settings where more money is available to be earned. Living in a rural setting is something one chooses not an immutable fact like skin color and by and large the harmful stereotype is spot on.