Well, to play devil's advocate, if all the tickets are free, and the attending gets to ignore all the ones that don't win, that could be useful.

FWIW, here's Lukas' writeup of his experience.

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4850894/

> if all the tickets are free

They're definitely not free; doctors have to evaluate what Watson says, and if Watson is wrong a significant fraction of the time, then doctors will spend a lot of time and energy barking up the wrong tree.

Eh, it's a tool. Some are useful, some less so. I imagine this would be a tier two kind of thing. It's an unusual case, what suggestions does the robot have? Maybe something obscure and easy to forget.

Doctors have to evaluate what patients say, too, and they're forgetful or clueless at least as often as anyone else. If the cost-benefit analysis favors having Watson (or whatever) around, great, if not, great, just another decision to make.

>nothing about any kind of controls or whether or not this was a one-off performance or if it would work at scale.

I thought the 20 million oncology studies Watson cross referenced was the control and I arrived at the opposite conclusion as you about the usefulness at scale.

For example, I immediately thought that under the Affordable Care Act, the number of patient Electronic Health Records the Government collects eclipses what Watson had to work with in this case so at scale the potential is huge. Admittedly, until reading your comment I did not consider there were likely a number of times Watson misdiagnosed (which seems obvious now). Therefore, it would be interesting to rerun against the Electronic Health Records the Government has collected so far and test if Watson could diagnosis correctly in less attempts than with the 20 million record it used (or what I call the control). Though at that point, I think it would be the patient that is the control.

The "control" for the actual (machine) learning may be the test set derived from the available data, but the control of a medical study must be real life (new data) and not old data used for the machine learning algorithm.

You don't need 100% new data to get an idea of performance. It's meaningful (indeed, common) to select some arbitrary percentage of the real-world data as a "training set" then measure the performance of the algorithm on the rest. It's not proof against overfitting, of course, but it helps.

right, but there's a reason we run clinical trials before accepting that something is superior to standard of care. It is really, really hard to manipulate a preregistered clinical trial (not that people don't try) and incredibly easy to manipulate train/test/validate results.

> After all, winning the lottery is a lot less impressive if you've bought a few million tickets

I don't know, I'd still be impressed, wouldn't you? Buying a few million lottery tickets is impressive by itself, both logistically and financially. And even still, odds. It's a lot of risk for anybody to dangle.

Shit, buying every ticket is even more impressive. (I grant your point, you just posed a funny metaphor as comparison.)

I don't think buying every ticket is more impressive, it is pretty much the definition of a losing strategy.

Tests like these are all about the statistics and without statistics this result - while important to this particular individual - is almost meaningless.

That's pretty cynical of you. Perhaps you should give credit where credit is due instead of attributing it to just PR. Watson diagnosed an illness that was undetected by her doctors. This happens all of the time and people die because of this or live in misery because of it. If Watson can prevent even a fraction of the rampant misdiagnoses and aid in the actual determination of the illness then it's in our benefit to use the information it provides.

It isn't cynical to insist on knowing the test's sensitivity and selectivity.

It's one thing to ask for additional information on how the test was conducted and another to automatically attribute it to a PR play that IBM likes to "milk every last drop of".

That's cynical.

Its not cynical if you've actually had any dealings with IBM and Watson.

Edit: And when the hell did cynical become a bad word in silicon valley? I've had a number of posters on here respond to things "oh, stop being cynical!" as if that was some kind of contribution...

I've got a word for people who are cynical.

Decent scientists.

Feynman would be rolling over in his grave...

There's a difference between cynical and skeptical, from google:

"A skeptic doesn't believe anything without strong reasons, which is why it is also associated with doubt, especially when something hasn't been experienced yet. Cynicism is believing the worst of something or someone. It has nothing to do with evidence. It is an outlook on life."

If cynicism is believing the worst in something or someone, then believing that this article is just PR is by definition not cynical, since it is not the worst thing one could believe.

You're replying to the wrong post I think. I was just correcting the misuse of the word cynicism, nothing else.

> It has nothing to do with evidence.

If you understand Bayesian reasoning, then we all have unjustifiable prior beliefs, perhaps due to innate personality, or due to our own experiences.

Calling someone a cynic is often just a way to bash someone who has different priors.

The better thing would be to ask where those priors come from, no?

> Its not cynical if you've actually had any dealings with IBM and Watson.

Maybe GP knows something you don't and he's talking from experience?

I'm sure the GP knows many things I don't. The only point of my post was the definition of the word cynical. It's probably fair to say that skepticism is an admirable trait while cynicism isn't. Feynman was a proponent of skepticism not cynicism.

The applicability of either moniker towards anyone in this debate, or any other, is up to the participants to hash out.

skeptical!=cyncial

> to automatically attribute it to a PR play that IBM likes to "milk every last drop of".

No, that's history. IBM has quite a series of Watson 'breakthroughs' that ultimately went nowhere, to the point where when I see 'Watson' mentioned in an IBM press release it automatically gets discounted if there isn't a part where the subject has some statistics to go with the inevitable hype.

I see it as IBM trying to stay relevant in a world that needs it less and less and that's why they focus on anything that will grab headlines.

Winning jeopardy, curing cancer, what's not to like about IBM?

http://webcache.googleusercontent.com/search?q=cache:GaC_y_3...

Cached link supplied because the original has departed.