This isn't a clinical study where the stats are everything. This is a real world situation where people are being profiled and it is legalized.

But I think the other side: to say that a coin flip would punish less innocent people, makes it a fair comparison.

> In U.S. v. Bentley, we see just how damaging the Harris decision really was. Lex, the drug dog that searched Bentley’s car, had a 93 percent alert rate. That is, when Lex was called to search a car, he alerted 93 percent of the time. He was basically a probable cause generator. His success rate was much lower, at 59 percent.

In other words, drugs were found in 55% of cars in which he was brought in.[1] The base rate for drug possession in a car is almost certainly a tiny fraction of that. So between the human judgment to bring in Lex, and Lex's sniffing, the police are able to find drugs at probabilities that are multiples of the base rate. In contrast, if you searched cars based on a coin flip (i.e. randomly), you'd expect to find drugs exactly at the base rate.

[1] Being successful > 50% of the time is pretty much the definition of probable cause.

Consider a drug sniffing dog at an airport, it sniffs all carry-on bags as people go through the line to be searched. If we find drugs even 10% of the time that the dog whines, it's amazing, because the actual distribution of drugs amongst travelers is minuscule, maybe 0.01 of 1%.

But if the dog is right 55% of the time? This is excellent probable cause for a search.

But by way of contrast, consider a drug-sniffing dog that is only brought in when a human handler suspects a traveller may be carrying drugs. If the handler is very good at their job, and optimizing for success, the handler may themselves have a 55% positive-positive rate. Meaning, 55% of the people they pick are carrying drugs.

Now if the dog whines 55% of the time they are given a suspect that has a 55% chance of carrying drugs, well, that is no better than a coin flip, it's garbage.

And of course, what if the handler is amazing at picking people who might be carrying drugs? For example, what if they have access to warrentless surveillance meta-data and are using the dog as an excuse for parallel reconstruction ("Your honour, I picked this person at random, and the dog whined, so I searched them"). The handler might have a near 100% success rate of picking people carrying drugs.

Now a 55% rate of whining is truly terrible for deciding whether the dog is any good.

Going back to the original case, even if we assume that every time the dog doesn't signal there wasn't drugs (so the dog never missed a drug), had the police just searched without bringing in the dog they would've had similar rates. So why isn't the police's 'I think he had drugs, can't give any good reason though' not probable cause if the success rate is nearly 55% when the expected success rate of searching everyone is under 1%?

There is some aggregate base rate, which is the percent of a random sample of motorists that are carrying drugs.

Then there is the rate of those who are actually carrying drugs within the a group that the police have pulled over, and suspect are carrying drugs, and have also presumably asserted their rights and refused a consensual search by the police and also do not have some other attribute (such as smell, items visible in plain view) that would provide probable cause.

You argue that "the base rate for drug possession in a car is almost certainly a tiny fraction of that" but in fact the base rate within this much narrower second group should be extremely high in comparison to the general population. And the problem is that searching that group, regardless of what you think of them, is illegal.

In order to make it legal you have to be able to show that you're going to do something specific that will provide probable cause.

Showing that the dog is anything short of a clearly reliable and accurate means of determining that a law is being violated means that the dog is not providing probable cause. A dog that alerts to drugs more 90% of the time when only 60% or so of the population in question has the drugs, has shown results that aren't sufficiently correlated to the data it is supposed to discern.

When that is true it is just a tool for violating laws that prohibit searches.

Yes, but that hurts your argument rather than helping it. The drug dog does not, by itself have to clear the probable cause hurdle. Rather, the totality of the circumstances has to yield probable cause: https://en.wikipedia.org/wiki/Illinois_v._Gates.

So say the base rate for carrying drugs is 10% in the population. The police, based on their suspicions, pull over a population of people where 40% are carrying illegal drugs. Then, the drug dog comes in, and he's barely better than a coin flip--60% correct. Out of 100 people stopped, he'll correctly identify 24 of the 40 carrying drugs, and incorrectly flag 24 of the 60 not carrying drugs. But overall the process will be right 50% of the time, versus only a 10% probability if you stopped drivers at random. That's probable cause.

For example, let's say that the dog is not sufficient probable cause. Consider the cases:

1) drug dog indicates + suspect driving erratically

2) drug dog indicates + "I just have a feeling"

Am I correct that 1) might be PC but 2) could not be, no matter how accurate that officer turns out to be in practice?

(In case 2, the dog's finding more drugs than the base rate would not validate its accuracy, as the results are being pre-filtered to have a very different distribution.)

So rationally, the cop's "spidey sense" is indeed evidence. The question I posed to you (and whoever's reading) was whether it counts as legal evidence. And it's my understanding that, no matter how good your spidey sense is, it does not become PC in a legal sense until it is articulable. As Thomas Moore said in A Man for All Seasons:

"The world must construe according to its wits; this court, according to the law."

https://en.wikipedia.org/wiki/Receiver_operating_characteris...

Now, what's the actual base rate? What proportion of cars on the road contain drugs? I'd wager it's well under 10%. Even if we say it's 10%, that means that the humans are taking the rate from 10% to 55%, and then the dog goes from 55% to 59%.

In other words, the dog is basically useless, except as a tool to make the searches legal. If not for the legal requirement, one could just as easily say that "between the human judgment to bring in the magic 8 ball, and the magic 8 ball itself, the police are able to find drugs at probabilities that are multiples of the base rate."

Apparently police are really good at figuring out whether cars contain drugs, but the law doesn't allow them to act on this directly. And apparently dogs are nearly useless at figuring out whether cars contain drugs, but the law allows their nearly useless indication to be used as the basis for action. This is rather backwards!

One wonders if we have the procedures we do because it gives a thin veneer of scientific respectability over a system that might otherwise appear biased, racist & otherwise distasteful even if it's statistically valuable.

There does seem to be a lot of "this is 'scientific' so it's good" in law enforcement. A lot of forensic techniques are just BS, but are accepted because they're "scientific." Or look at the recent article posted to HN about how polygraphs are nonsense.

Of course, it's the job of the police and the courts to come up with an answer, so we should be able to let them do it. But it seems they can't be trusted to reason about it well.

If all you need is your own suspicion, it could be easy to let one's bias against purple people run rampant, and search every purple person's car and nobody else's.

If you have to call in a dog which you believe to be effective (even if it may not be) then you may only go to that trouble when you actually have some legitimate suspicion, rather than just going based off your anti-purple racism.

No, it isn't. 'A common definition is "a reasonable amount of suspicion, supported by circumstances sufficiently strong to justify a prudent and cautious person's belief that certain facts are probably true"' https://en.wikipedia.org/wiki/Probable_cause#cite_note-2

I would argue that if you roll a 100 sided dice, my belief that the result is over 45 is not 'prudent and cautious' but 'optimistic'.

The point of that analogy was to create a situation in which someone is right 55% of the time and see how the language "prudent and cautious" feels in those circumstances.

Whether that means rolling over a certian number on a die, or rolling a specific number on a very strange weighted die is irrelevant.

Consider, instead, something like guessing whether someone has had cancer. Only 8.5% of the population has ever had it. If you can guess randomly, you'll be right about 8.5% of the time. If you can guess correctly 55% of the time, you're doing far better than random guessing.

The implicit assumption of the "probable cause" concept is that the base rate is low. I.e. only a small percentage of people have illegal contraband. If a police officer is able to guess that someone is more likely than not to have contraband (i.e. more than 50% correct), then he must be acting on some concrete clues, because he's doing far better than if he just searched people randomly.

"The problem here is that invasive searches based on no more than a government official’s hunch is precisely what the Fourth Amendment is supposed to guard against."

Due to the high alert rate, we actually have a pretty good idea of the base rate of drugs in the situations that this dog is being used in (~50-60%).

The question is whether the dog is actually providing probable cause when the dog has such a high false positive rate.

The assertion of the article is specifically that drug dogs are being used and trained by police in ways that allow them to be be used as "a probable cause generator".

"The Supreme Court originally gave its imprimatur to drug dog sniffs because when used properly, the dog’s finely tuned sense of smell can detect drugs and their absence with incredible precision. But that caveat — when used properly — is critical. If a drug dog isn’t eliminating any innocent people, if it’s validating the suspicions of the police nine out of ten times resulting in searches for which up to half or more of the suspects are innocent, then descriptors such as finely tuned and incredible precision no longer apply."

Thus the base rate that we should be judging the dog against is not the base rate in the general population, but the base rate in the dog's tested sample. We need to incentivize the cultivation of higher levels of accuracy that the dogs are capable of providing.

Now if you take the 50% of people you guessed to be positive for test, only 4.25% will be true positives, indicating 8.5% accuracy rate; if you take the 50% people you guessed to be negative for test, 45.75% will be true negatives, indicating a 91.5% accuracy rate.

But your overall accuracy rate is 50%.

edit: Seriously? A downvote for this comment?

I downvoted your comment because the argument you are attempting to make is incoherent. I'm sure that you understand what you are trying to say, but as written your comment does a poor job of expressing that point of view.

If have a procedure for guessing the number rolled on a 100 sided die, you are choosing between 100 different possible guesses. That is nothing like a dog guessing if a person has, or does have drugs (two possible guesses). Attempting to compare the accuracy rates between those two cases is major statistical error.

I thought this difference would be obvious to most readers of HN and was thus surprised to see such a simple statistical error being made. I did not think that verbose explanation would be needed once the error was pointed out.

But I don't think "prudent and cautious" can quite be boiled down to just a certainty of outcome. Risk/reward plays in a lot. If we take "cautious" to mean a medium degree of risk aversion, then I don't think a cautious person would risk their year's wage on the chance to double it, even with a 55% certainty of success. However, I think they would risk $10 for a 55% chance of getting $20.

So what is at stake? What's the cost of an invasion of privacy?

Because of civil asset forfeiture, the base value is however much cash happens to be in the car, assuming it's enough to be "suspicious" (how just having cash is suspicious, I don't know). Most people don't have large amounts of cash in their car, but those who do have, well, a large amount.

The other quantifiable loss to a car search is time. Based on a quick search, it seems the average hourly wage of americans is $25. Wage isn't really the best evaluation of time, but it's easy. That covers the time wasted, but not the losses due to being late to something. A car search could make someone miss an appointment, or even get fired — or nothing.

I'm going to estimate that the sum of how much the average person stands to lose to asset forfeiture and time is about $100. That's of course not even considering the emotional impact of having your privacy violated, or the potential non-legal consequences of having a car search, say having an affair exposed. I'll try to give a personal answer to that question. How much would someone have to pay me for me to be okay with them tipping of the police that I have drugs in my car? Well since I'd be able to plan for it, missing appointments and asset forfeiture are hardly a concern, but time is. I suppose someone could hire me to stand around and be bored for an hour for $40. I'd let them tip off the cops for say, $600. Since that's how much it would take to make it worth my while and not really just meet the baseline of acceptable, I'll cut off a bit from that, say $60. And then subtract the $40 for my time. Adding in the estimated tangible cost, and that's $600 again.

So let's go with $600 as the cost of an erroneous search. Now we need the value of a successful search.

It's hard to quantify (I'd argue the value is negative because of the questionable value of the war on drugs and the cost of incarceration), but for this purpose, I'll use bounties. A quick search turned up a really bullshit looking page that mentions a county police department offering $1000 cash to anyone who turns in a heroin dealer. While the value to the county is going to be higher than the bounty they offer, I'm going to assume that since this county is offering a bounty in the first place, they value it slightly higher than other municipalities. So let's go with $1000 as the value of catching a drug dealer.

Risking $600 for a 55% chance of $1000 isn't just not cautious, it's a losing game.

*

So what would I consider cautious? Risking $1000 for a 60% chance of doubling it? I'd characterize that as "cautiously optimistic". 65%? Same. 70%? Sure I'll call that "cautious". Obviously different people will have different ideas of what "cautious" means.

But by my definition, and my value estimates, I'll say the certainty required for probable cause for searching a car with drugs is 85%.

The main gist of which is the difference between 'concrete' and 'statistical', which seems to be very different from the way many of use think about 'reasonable'.

The article certainly indicates that you don't need 85% certainty to have probable cause. While officers can have pretty high accuracy rates to their guesses, they need a 'concrete' piece of evidence to point to so they can have probably cause. Drug dogs are being used to create this concrete reason, but this has more to do with their perceived reliability than their actual effectiveness reliability.

My biggest concern with the points that this article raises is that it seems like the dogs are being trained to reliably provide probable cause rather than being trained to provide reliable probable cause.

So the article is correct as written. About four out of every 10 roadside searches conducted on the basis of a dog alert are groundless. This outcome is nearly indistinguishable from a coin flip conducted by the officer who has already flagged the car as a suspect vehicle.

Out of the 100 the officer picks the dog is choosing 93%... but is only right 55%... the dog correlates to the officers choice and both are only right about half the time

That becomes scary when you're talking to a doctor about a test that returned a positive result. Does that doctor know how likely it is you actually have what you tested positive for? Are you going to get treatment for this thing?

Gerd Gigerenzer has a nice book about this: http://www.amazon.com/Reckoning-Risk-Learning-Live-Uncertain...

Here's one example about breast cancer testing: http://imgur.com/zO4zkl4

I think that articles like this one, if they were better, could move things (even if just a little bit) in a positive direction by educating the public. Unfortunately when they are as poorly written as this I think they do the exact opposite. Oh well.

;-)

It's important to keep orthogonal issues separate so you don't cross contaminate and introduce unintended outcomes.

Well, you set up a straw man, so the rules of logic had already been implicitly suspended in that branch of the conversation.

> the same questions apply on searches for guns, stolen property or any other contraband

No, they don't. Dogs can't generally sniff out guns or distinguish between stolen and non-stolen property, and so dogs can't be used as a pretense in those situations. Using a dog to sniff out stolen property (unless that stolen property happens to be a salami) would make it obvious that the dog is just a pretense.

> It's important to keep orthogonal issues separate

Indeed.

So let's review: the problem is that although dogs can sniff out drugs, unless some very strict protocols are followed, they generally don't sniff out drugs and instead just tell their handler what they want to hear, i.e. that there are drugs (though, of course, the dog has no idea that this is how its actions are being interpreted. All it knows is that if it signals, it gets a pat on the head.)

As a result, dogs are not an effective crime fighting tool, but instead a de facto pretense for doing an end-run around the fourth amendment. The same effect could be produced by other means, e.g. flipping a biased coin, but that would defeat the purpose because that would make the bias obvious, which is exactly the rhetorical point that the original author wanted to make.

Do you really want to disagree with any of that?

If you only consider the 930 cases where there is verification, you do come up to 59% accuracy.

Even using your generous assumption of the 70 cases being true negatives, the dog only come up to 62% accuracy.

However my reading is the the 'coin flip' is being used because the dog's accuracy is near false positive rate 50% and they are trying to compare the dog's accuracy vs. a randomly accurate instrument to illustrate the following point to non statisticians:

"If a drug dog isn’t eliminating any innocent people, if it’s validating the suspicions of the police nine out of ten times resulting in searches for which up to half or more of the suspects are innocent, then descriptors such as finely tuned and incredible precision no longer apply."

Care to cite that?

Because that means that in a neighborhood where 51% of the houses contain narcotics, the police have probable cause to search every single house.

>We do not know exactly what the phrase “probable cause” means, in strict numerical terms. We do, however, know what it does not mean: “probably.” That is, probable cause does not—in the context of Fourth Amendment law—mean that the police must have evidence sufficient to conclude that a suspect is probably guilty or that she probably has evidence of a crime hidden inside her home.

I didn't read the whole paper, but the gist of it seems to being exploring the difference between 'concrete' and 'statistical' and evidence of effects (with a non-conclusive, but general indication that the supreme court leans towards 'concrete' over 'statistical')

Basically if you have a 3 people in a car that contains drugs and thus know 1 of them is guilty, you have probable cause to arrest all of them. (Concrete)

However, if you know that 1/3 of the black people in a certain neighborhood posses drugs, that does not mean you have probably cause to arrest all of them. (Statistical)

Thus my argument is that the low accuracy rate of the dog vs. the baseline of the searched people (which in this case is only known because of the high alert rate) should not allow it to constitute 'concrete' evidence of probable cause.

Additionally, the overall success rate of the dog/officer combination vs. the general population baseline constitutes 'statistical' evidence which is also not sufficient for probable cause.

Generally, my argument would be that allowing the dogs to serve as a final determination of probable cause without requiring high accuracy rates leads to unnecessary harm to the general populace in the form of the unwarranted searched that could be prevented by encouraging effective training and usage.

The problem is innocent people who get searched illegally. Being sent to jail is not the only bad thing which can happen to people; being searched without probable cause is inherently unacceptable without regards to the results of the search.

Therefore the standard of evidence for sending people to jail is higher than the standard of evidence to search people.

Therefore the standard of evidence for probable cause is lower than the standard of evidence for a criminal conviction.

Not from a Constitutional law standpoint. They are both violations of the subject's civil rights, when the search is based on a pretense, a pretext, or a coin flip.

He is however missing the point in that he is bringing up a separate standard of evidence that has no bearing on the issue being discussed. Being searched isn't bad just because it can lead to you going to jail.

Perhaps, but my point is that neither standard is being met. If the officer didn't have PC before calling in a dog with the performance history described in the article, then he doesn't have PC afterwards either.

It is not OK to violate one aspect of the Fourth Amendment just because it appears (in someone's opinion) to call for less stringent adherence than other aspects.

So we'll need to agree to disagree on this one. I suspect we're coming from two very different places.

My point here has nothing to do with the war on drugs. The same standard applies if law enforcement is searching for drugs or guns or stolen property.

You should ponder this some and then consider changing your point of view so that you can be correct more often in the future.