I think that secret shopping is a great idea for forensics, but I worry that in areas of criminality, just pushing fake requests over to forensics won't work unless it's accompanied by the usual pleas from the cops and DAs actively working the cases to which those data are associated.

Similarly, a rush order is a rush order, but a rush order with a group of people carping on how it's a rush order tends to get more rushed. When the queues are effectively bottomless, stamping 'rush' on an order without the associated pressure seems likely to relegate its status to regular.

Maybe submit multiple controls in the same request.

Feynman gave a talk once which included a by-the-way remark like "this is not a scientific age". It stuck in my mind because of course that wasn't the common view of the 1950s or 60s when he said it. The obvious reason to say it is that some people still consume astrology, etc. But I came to realize that this is the sort of non-scientific-ageness that stands out. About all sorts of important things in daily life we don't collectively act as if we're curious about the truth and ready to do what it takes to learn it. If this were a scientific age then not routinely checking practices like this would stand out. Everyone would wonder if they had something to hide.

Programming was less routinely scientific like that in my own experience earlier in my career when automated testing was less common.

I think the Feynman talk was the one that turned into his posthumous book The meaning of it all, though I'm not sure.

>That technique is blind testing with published results. 10% of all submissions to the testing lab would be test samples -- a mix of a bunch of known negative samples (from individuals completely unrelated to the crime in question) and a few known positive samples (perhaps from someone already convicted with overwhelming evidence). The lab would not be told which samples were tests and which were "real", and the test samples would be evenly distributed (10% of the submissions from each department; 10% of the "rush" submissions, etc.). Finally, the results reported on these test samples would be made publicly available.

And not only would it let you weed out the pseudoscience but it would let you weed out the labs that are practicing the real science well from the ones that are sloppy (which in my opinion is an equally large problem).

That, and the labs should not be a branch of law enforcement or prosecutors' offices. They should be more like a weights and measures department, and be at the service of both prosecutors and defendants.

These labs are private businesses. Prosecutors and police can choose which ones to use and they will use, and they'll likely make that choice based on who gives them the results they want rather than the results that are accurate.

In Massachusetts, there is a state police crime lab: https://www.mass.gov/orgs/massachusetts-state-police-crime-l...

It is a department of the state police, and, obviously enough, is structurally incentivized to make state police performance look good.

The Massachusetts state crime lab is an actual madhouse, and the state government is shockingly corrupt. The following article is one of the more detailed descriptions of how the lab scandals were managed

https://www.rollingstone.com/culture/news/did-falsified-drug...

>and the state government is shockingly corrupt

Shocking to the blind and deaf maybe. If it's under the executive branch it's a dumping ground for political appointees. That's just how it works in MA. More law, more bureaucracy to enforce it, more dumping, more bad outcomes, more laws, more bureaucracy...

I've lived in three other states and none of them were as bad.

So there is incentive for corruption with both public and private labs. The solution will require something far more drastic, which I'm not sure the population is willing to support given how many people don't like to use science when it disagrees with their emotional stances. As long as the population wants tough on crime stances, and are willing to dehumanize anyone charged (much less wrongly convicted), then the system is giving them what they want. So how can we change the voters so that they demand a more just legal system?

that's largely not what motivates law enforcement's choice (prosecutors aren't generally the ones choosing the testing laboratory; it isn't what motivates them, either); keep in mind that contracts have to go through the accounting/legal department.

they care about cost, and turn around time. availability to answer their questions, and some level of treating them with basic respect are up there, too (by "basic respect" i mean avoiding things like "i don't have to explain shit to you, you're a dumb cop and i'm the fancy scientist").

if you're regularly telling the DREs that they're wrong, you're going to have some arguing/explaining to do, but it is unlikely to lose you the contract.

the labs that go rogue usually do it for stranger self-motivated reasons than "gotta give the cops the exact results they want"

> These labs are private businesses

Yes, these ones are, but the ones that are part of law enforcement offices (particularly, the FBI’s lab) have been even more notorious problems (like the complete junk science of hair comparison) driven by a clear conflict of interest.

In the case of the Motherisk program, the discredited physician was affiliated with a local university hospital.

accredited labs already have to do some of that. the organizations chosen (college of american pathologists, for example) to administer the tests are fucking garbage.

so it's a bit harder to get this kind of validation going than just deciding to test the labs.

> ...a mix of a bunch of known negative samples (from individuals completely unrelated to the crime in question)

haaah. you have no idea how many people are on something. getting known blank samples is nontrivial.

> Although this simple, effective, and cheap method would completely transform forensic practice, it will not be implemented. The reason is incentives. The parties involved in forensics are the police or prosecutors and the labs. Both have an incentive to identify the guilty.

there are independent labs that work for prosecution, defense, civil cases, etc etc every form of case you can imagine, and basically throw away all case information for the middle part of the testing process, so the folks who could hypothetically swing things one way or the other don't even know which way they'd be throwing cases if they tampered with it.

If someone had enough money, could they conduct this independently of the government parties?

many of the government labs won't take samples from private parties, and the ones that do frequently don't advertise the fact.

the private labs, sure, you could spike (or not) some blood samples and pay for them to be tested.

It is different in Canada. These kinds of things will be fixed because the SCC will rule that the laws enabling it are without force and the government will get a year to fix it. It

This was a failure of the court itself to enforce proper standards of evidence. For once you can't blame parliament.

It is really sad that it took over 30 years and the person is still working, albeit in Israel. But wouldn't the Evidence Act(provincial and federal) set the standards that allowed this?

The Evidence Act can further restrain the court but in a scenario that is ambiguous the court will rely on case law (e.g. law the court itself made up based on prior decisions). Should parliament disagree with the decisions of the court they may amend the law to force a change in future trials.

We're used to hearing about the supreme court restraining the actions of parliament by declaring something unconstitutional but in actuality the parliament exercises far more control over the court than vice versa. Even in the case of the court declaring something unconstitutional parliament could overrule them by changing the constitution itself (although that is extremely unlikely given the political climate).

Something could trivially be done to the same effect. A private party, say an investigative journalist, could submit a number of samples and report on the results (perhaps that happened in this case and that's why we know about it?). But to keep costs under control, instead of testing like this on every lab on a regular basis, that private party could randomly select labs for testing. While they're at it, they'd probably prefer to select labs with complaints, rather than a random sample. And at that point, you have something resembling the system we currently operate under, which has--to be fair--been completely successful (if a bit later than ideal) in identifying this problem lab. But as taxpayers, is it worth instituting a new regulated program over this essentially free solution?

Besides, this seems like a case of closing the barn door after the horse has escaped. Successfully identifying a problem lab--albeit tardily--is a signal the system is working; not that it needs a complete overhaul. Rather consider what completely unrelated problems would benefit from regulatory oversight at a higher return on investment. Focus on those, not on a solved problem.

You're assuming there aren't any other problem labs that haven't or won't be discovered. How expensive are these tests? How much of the police's budget is it? Surely you know these things if you are talking return on investment, because you must know these things to actually know what your return on investment is.

25 years is a long time to be improperly ripping apart families. Is there a point at which it becomes a problem? Death tardily solves all your problems after only 80 years, so I assume somewhere between 25 and 80.

I'm not saying it isn't a problem, only that we need to pick and choose where our dollars go. Is this really a better investment than, say, increased funding for public broadcast programming that might discover this kind of thing in a less directed way? That way you don't end up with yet another obsolete program hanging around costing money when this problem is no longer relevant.

And I think the onus is on the suggester to provide data to prove ROI.

The essentially free solution cost 25 years worth of bad results.