News for hackers. Stuff that matters.
Re:Court rules drug dog barely more accurate than a coin fli... by fiatmoney on Friday August 07, 2015 @12:39PM (#10023416)
The dog isn't even "as accurate as a coin flip". Given a 93% alert rate, regardless of the subsequent true positive rate, information gain from an alert is approximately zero.

Edit: "accurate as a coin flip" is a bad construction, but I wouldn't exactly expect better from a newspaper. The information gain from a device that activates randomly 50% of the time, 0% of the time, and 100% of the time are all the same, which is to say zero.

Re:Court rules drug dog barely more accurate than a coin fli... by gburt on Friday August 07, 2015 @12:53PM (#10023502)
This isn't being fair. I don't really think the post title is fair either. That 93% isn't comparable to the 50% you're probably thinking for the coin flip, or even the 0/100% mentioned in your edit.

---

93% alert rate means:

93% = false positive + true positive

7% = true negative + false negative

---

We're given "His success rate was much lower, at 59 percent." Assuming that means that 59% are true positives, we know the false positive rate is 34%.

Without data on how many passengers are carrying drugs, we can't say anything about the rest of that result. A plausible world where the drug dog may be valuable is when he unquestionably rules out that 7% of the population that he doesn't alert on, as in false negative = 0.

I believe the reason this number is so high (59%) is that the officer has already used his suspicion to determine filter out most people. This makes the value of the "ruling out" plausibility even more valuable: the drug dog reduces searches compared to trusting the cop.

Ideally, they'd provide the receiver operating characteristics curve [0] or the confusion matrix [1].

[0] https://en.wikipedia.org/wiki/Receiver_operating_characteris... [1] https://en.wikipedia.org/wiki/Confusion_matrix

Re:Court rules drug dog barely more accurate than a coin fli... by nickff on Friday August 07, 2015 @01:00PM (#10023541)
The real question is whether the dog is better than the officer at guessing who has drugs. If the dog is much better than the officer, it would suggest that a dog 'alert' should be given more weight than the officer's guess. If the dog is no better than the officer, the dog is useless.

I say this because it is possible (and indeed likely) that the officer is also better at guessing who has drugs than a coin toss.

Full Disclosure: I have never used or trafficked contraband, but I do not support random searches or the use of drug dogs in public without a warrant, regardless of their effectiveness.

Re:Court rules drug dog barely more accurate than a coin fli... by LordKano on Friday August 07, 2015 @01:09PM (#10023578)
If the dog is much better than the officer, it would suggest that a dog 'alert' should be given more weight than the officer's guess. If the dog is no better than the officer, the dog is useless.

A very big problem is that drug dogs will alert when the handler gives it subtle clues. This is done to get probable cause for a search when there is nothing more than an officer's hunch or just a desire to hassle a particular individual.

Re:Court rules drug dog barely more accurate than a coin fli... by mejari on Friday August 07, 2015 @01:16PM (#10023620)
This is something I see claimed a lot. I'd be very interested in any information you have about this. I find it very unlikely this kind of training is part of the official training, so I'm imagining it's some kind of unintentional conditioning? Or is it done on purpose by the officers?
Re:Court rules drug dog barely more accurate than a coin fli... by lukeschlather on Friday August 07, 2015 @01:31PM (#10023714)
This is covered in the article, and it's quite straightforward. The handler was giving the dog treats for alerting, rather than giving treats for finding drugs.

The primary trouble I see is that there is no penalty for re-training dogs like this.

Re:Court rules drug dog barely more accurate than a coin fli... by mejari on Friday August 07, 2015 @02:35PM (#10024136)
The article only discusses that the handler gives the dog treats for alerting, it doesn't go into the motivations about that, whether they're because the officers aren't considering the Pavlovian conditioning involved and are accidentally reinforcing such behavior or if it's intentional.
Re:Court rules drug dog barely more accurate than a coin fli... by kd0amg on Friday August 07, 2015 @02:50PM (#10024245)
Does the motivation matter when we're talking about whether it establishes sufficient justification for searching someone's car? After a clear flaw is pointed out, a response of "We didn't mean to ruin the statistical power of this test" wouldn't do much to restore my confidence in its results.
Re:Court rules drug dog barely more accurate than a coin fli... by mejari on Friday August 07, 2015 @03:13PM (#10024385)
I was specifically responding to this comment

>This is done to get probable cause for a search when there is nothing more than an officer's hunch or just a desire to hassle a particular individual.

that claims that this is done purposefully. That's what I was wondering about, not the overall efficacy of drug dogs.

Re:Court rules drug dog barely more accurate than a coin fli... by radiorental on Friday August 07, 2015 @01:35PM (#10023738)
You can watch this 6 minute video of a stop, I've linked to where the officer was caught triggering the search

https://youtu.be/w-WMn_zHCVo?t=3m35s

It happens.

Re:Court rules drug dog barely more accurate than a coin fli... by mejari on Friday August 07, 2015 @02:31PM (#10024117)
Thank you. It is kind of the videographer's assertion that "Check here" is a command to trigger instead of just a command to check there, however, isn't it?
Re:Court rules drug dog barely more accurate than a coin fli... by radiorental on Saturday August 08, 2015 @07:38PM (#10028977)
Yes but no drugs were found and the 'scratching at window' was the trigger for search. As the op article indicated - four legged search warrants.
Re:Court rules drug dog barely more accurate than a coin fli... by mejari on Sunday August 09, 2015 @05:11AM (#10029922)
That doesn't really address my comment. It's still just an assertion that that is an intentional trigger. The op article similarly makes this assertion but there is no supporting evidence that I saw.
Re:Court rules drug dog barely more accurate than a coin fli... by jlarocco on Friday August 07, 2015 @01:40PM (#10023761)
It's not intentional, but it's a real thing:

http://www.ucdmc.ucdavis.edu/welcome/features/2010-2011/02/2...

Re:Court rules drug dog barely more accurate than a coin fli... by mejari on Friday August 07, 2015 @02:38PM (#10024161)
So it's less an issue of police officers intentionally triggering things in order to harass individuals like the original commenter said and more of unintentional training. Thank you for the link!
Re:Court rules drug dog barely more accurate than a coin fli... by LordKano on Friday August 07, 2015 @03:44PM (#10024570)
One can call it plausible deniability.

There are many examples that have been caught on camera, I see that another commenter posted a link to one of them on this thread.

When you see things like the handler returning to a previously cleared spot and indicating that the dog should check it again; three or four times until the dog alerts, you can assume that the handler has a severe short term memory problem and he just forgot that the dog had previously checked the area and failed to alert. I think it's more indicative of the officer hinting to the dog that "this is where I want you to alert" and their training reinforces the idea that if the dog does what's expected, there will be a treat.

Re:Court rules drug dog barely more accurate than a coin fli... by mejari on Friday August 07, 2015 @03:57PM (#10024630)
That's why I was asking about if there was any information about this, or any studies/investigations done, since people can go back and forth forever with "Well, did they already check that part? Is it intentional? Was the dog triggered?" Etc.. etc... It definitely seems like an area ripe for investigation, I just haven't seen any.
Re:Court rules drug dog barely more accurate than a coin fli... by marincounty on Friday August 07, 2015 @01:58PM (#10023907)
Antidotial evidence: I have never had a dog that didn't know my moods, fears, apprehensions, etc.? I don't think being called "Man's best friend" is a worn out cliche?

My last dog picked up on so many of my personality traits/tells-- I felt quilty.

(My cat--whole nother story.)

Re:Court rules drug dog barely more accurate than a coin fli... by mejari on Friday August 07, 2015 @02:36PM (#10024146)
Very true, I guess my question is about the intentionality of it. I doubt you intentionally trained your dog to respond to your moods, but he/she did it anyway. The claim here is that officers are doing this intentionally in order to get around the law of probable cause. I'm interested in cases where this has been shown to happen.
Re:Court rules drug dog barely more accurate than a coin fli... by gburt on Friday August 07, 2015 @01:02PM (#10023551)
I have no opinion on the legal and ethical issues, I was commenting only on the classifier. I don't think your first question is appropriately described as "the real question" though, as one of the coolest results of machine learning is that there are techniques for combining multiple bad classifiers [0] in to one really good one.

Even more so, I think that is actually what is happening here. An officer is acting as a tier 1 filter, and then the drug dog is acting as a tier 2 filter, further reducing the number of false positives, which in a "legal and ethical" sense, we generally feel are worse than false negatives.

[0] https://en.wikipedia.org/wiki/Ensemble_learning

Re:Court rules drug dog barely more accurate than a coin fli... by nickff on Friday August 07, 2015 @01:14PM (#10023609)
>" An officer is acting as a tier 1 filter, and then the drug dog is acting as a tier 2 filter, further reducing the number of false positives, which in a "legal and ethical" sense, we generally feel are worse than false negatives."

If the officer is acting as a filter, then it seems to me like this is a search, and not allowable, because it would be subject to all the usual objections to random searches. You might be right, but if that is the only way for these searches to be better than the officer's guesses, it seems that drug dogs would not be allowed to 'sniff' in public without warrants.

Re:Court rules drug dog barely more accurate than a coin fli... by jeremysmyth on Friday August 07, 2015 @01:09PM (#10023580)
It's in the article. The 93% in that study includes a 59% true positive (of 93%, presumably) success rate. We can't know how many true or false negatives there are because those searches aren't actually carried out, so out of every hundred cars stopped for a dogsniff, 55% (59% of 93%) are found to be carrying drugs.

More worryingly, that also means that out of every hundred cars stopped, 38 are wastefully searched by police (after getting probable cause from the positive dog alert) without finding drugs. From the article, a search takes about 45 minutes and carries a bunch of other risks for occupants of suspected cars (e.g. civil forfeiture, overenthusiastically trigger-happy LEOs), so this is pretty bad situation, not just in wasted time and cost, for both the car occupants and the police force.

One slightly positive thing I'll take from this: Over 50% of cars that are dogged have drugs, and I sincerely doubt that over 50% of cars (on average) have drugs, so in this particular study it appears that police have a better than random chance of guessing which cars have drugs. It's not a very positive thing though, because civilized countries that operate under the rule of law cannot permit law enforcement to operate on guesswork. This is underlined by the fact that the article mentions that the false positive rate rose to something like 80% for latino drivers.

Re:Court rules drug dog barely more accurate than a coin fli... by gburt on Friday August 07, 2015 @01:13PM (#10023605)
Sorry - I edited my post multiple times after posting it to clean it up and increase quality. Some of your response (especially the first paragraph) is now confusing. That was my fault.

I agree with you mostly - especially on the discussion of what "probable cause" should be defined as and what constitutes a "random" search, as just an example of some of the ethical issues of interest here. That is a debate to have and even one I think this community is well equipped to discuss, but it was not something I was trying to comment on.

I was only making the point that it wasn't fair to jump on the 93% as though it was representative of the problem. I am not taking any ethical or defensive position of drug dogs here.

Re:Court rules drug dog barely more accurate than a coin fli... by brobdingnagian on Friday August 07, 2015 @01:07PM (#10023573)
The dog has low precision and high recall. The dogs are capable of high precision and high recall, but aren't being trained to do that because other incentives are at play.

I found it impressive that the dog that alerts 93% of the time was correct 40% of the time. That means the police are doing what seems like a good job of identifying targets that actually have drugs. With those kind of odds, you'd probably want to take a closer look at everyone just to make sure you aren't letting people with drugs get away (false negatives).

Re:Court rules drug dog barely more accurate than a coin fli... by mikeash on Friday August 07, 2015 @01:12PM (#10023595)
It's plausible that the 7% is all true negatives. It's also plausible that the 7% is carrying drugs at the same rate as the 93%, and the drug dog is doing nothing at all. If that is the case, then flipping a biased coin would be just as effective as the drug dog.
Re:Court rules drug dog barely more accurate than a coin fli... by okasaki on Friday August 07, 2015 @01:08PM (#10023574)
Erm, let's say 10% of the people on whom drug dogs are called have drugs.

Then for 1000 stopped, you'll have

0.59 * 0.1 * 1000 = 59 true positives and ...

0.34 * 0.9 * 1000 = 306 false positives.

That's pretty terrible.

Re:Court rules drug dog barely more accurate than a coin fli... by harryh on Friday August 07, 2015 @01:16PM (#10023619)
Your math is wrong.

The article indicates that of all the people the dog indicated had drugs 59% of them actually did. So for 1000 people whom drug dogs are called on

930 the dog indicated had drugs. Of those 549 were correct and 381 were false positives.

Re:Court rules drug dog barely more accurate than a coin fli... by merrywhether on Friday August 07, 2015 @01:40PM (#10023766)
I think the coin-flip analogy comes from the idea that once an officer thought you had drugs, he/she was about 55% right (not accounting for false negatives). The dog alerted most of the time, thus allowing for search to confirm or deny the suspicions. Thus the dog gave no real information and the officer was the coin, but the reporter conflated these success rates.
Re:Court rules drug dog barely more accurate than a coin fli... by protonfish on Friday August 07, 2015 @12:48PM (#10023473)
If I had a coin that showed heads 93% of the time, I'd think it might be rigged.
Re:Court rules drug dog barely more accurate than a coin fli... by DSMan195276 on Friday August 07, 2015 @01:21PM (#10023649)
IMO, it doesn't seem remotely comparable to a coin flip because the odds aren't 50/50. The dogs are only brought out in certain circumstances when they think a positive might be possible, meaning that lots of people who would be negatives aren't tested and thus not included in that 93%. The cases making up the 93% only consist of people whom they thought may get a positive, meaning the sample is biased to having lots of positives.

I'd compare it to the fact that lots of very hard certification tests actually have fairly high pass-rates (80% or above), simply because people who don't think they could pass don't take the test to begin with - The 80% pass-rate is not representative of the average person's chances of passing the test, nor the difficulty of the test itself.

What's really important is the percentage of those positives which were true positives, vs. false positives. In this case, you would expect the number of positives to be high - The number of true positives vs. false positives would tell you how accurate the dog actually was in the positive case. If this number is also high, then the dog is doing a reasonable job. Of course, to derive a real argument from this, you'd need to know the number of true and false negatives as well, but obviously we don't have that information.

Re:Court rules drug dog barely more accurate than a coin fli... by slapresta on Friday August 07, 2015 @01:42PM (#10023783)
Sure, because a coin is evaluated on its capacity to map to an even distribution of outcomes. Drug possessions, however, are almost surely not an even distribution. If, say, 95% of the people that the dog evaluates have drugs, then raising an alarm 93% of the time is probably pretty good.

In other words: If we wanted a drug dog to be as close as possible to a coin flip, we'd just flip a coin instead.

Re:Court rules drug dog barely more accurate than a coin fli... by JoeAltmaier on Friday August 07, 2015 @12:41PM (#10023436)
Be fair. False positives and false negatives are very different things. A drug dog that finds drugs even sometimes, is still useful.

Imagine if it was bombs they were sniffing for. We'd be happy if they found even one more bomb.

Re:Court rules drug dog barely more accurate than a coin fli... by gruez on Friday August 07, 2015 @12:43PM (#10023447)
>A drug dog that finds drugs even sometimes, is still useful.

Why even bother with the dog? why not just do random searches? Oh wait, because that would be unconstitutional. The dog exists solely to justify otherwise unconstitutional searches.

Re:Court rules drug dog barely more accurate than a coin fli... by JoeAltmaier on Friday August 07, 2015 @12:47PM (#10023470)
Ignoring the fact that false positives don't preclude real positives...it works. People are fond of cute doggies, and often their resentment evaporates.
Re:Court rules drug dog barely more accurate than a coin fli... by zajd on Friday August 07, 2015 @01:03PM (#10023556)
[citation needed]
Re:Court rules drug dog barely more accurate than a coin fli... by elektromekatron on Friday August 07, 2015 @01:13PM (#10023601)
False positives only don't preclude real positives if time and resources are not constrained. Time and resources are always constrained.
Re:Court rules drug dog barely more accurate than a coin fli... by pavel_lishin on Friday August 07, 2015 @01:03PM (#10023553)
Then we should simply search 100% of the bags.

Or maybe just 93%.

Re:Court rules drug dog barely more accurate than a coin fli... by elektromekatron on Friday August 07, 2015 @01:02PM (#10023549)
If it was explosives and a dog got a 93% alert rate irrespective of there being any explosives, then that would be useless and nobody in their right mind would use that dog, as it would have them search 93% of everything.

At that point you might as well give up on faffing about with a sniffer dog and just search the remaining 7%.