Different surgeons in different areas treat different kinds of patients. It's hard to accurately measure anything in a meaningful way that should influence decision making. To use your example, surgeon Z may also perform procedure Y but has (N-5)% of patients reporting relief and (M+5)% of patients reporting post-op complications. However, surgeon Z works at a community hospital and treats a poorer patient population with more co-morbidities. Can you really say if surgeon X is better than surgeon Z?

And it shows up all the time in things like the Consumer Reports hospital rankings and the like, where hospitals with particularly uncomplicated patient populations come off looking like they're the best hospitals.

Doctor's historical success rate exceeds expected success rate on average => Good (or lucky) doctor.

Genetics is oversimplified to non-physicians. It's cool that we can diagnose and predict the likelihood of getting Huntington's disease using our knowledge of genetics, but extremely few diseases are this simple. There are huge swaths of the human genome that we don't understand but are likely playing some important role in the regulation of other genes and diseases. We are nowhere close to being able to look at a patient's genome to predict anything useful outside of a handful of exceptions.

Patient histories are honestly often garbage—I say that as a physician. I look through dozens of patients' charts every day, and there are constantly errors, incomplete documentation, and fragmented records across multiple institutions. Just last week I read a chart for a patient who had a documented hysterectomy from years ago. The brand new CT scan I saw showed a perfectly normal uterus. Once something goes in a patient's history it's nearly impossible to correct or remove. If some doctor from ages ago said the patient is allergic to medication X, but the patient denies it, what do I do? Usually, we opt to leave the allergy listed out of fear of the consequences if the patient is wrong.

I had one doc I had only met remotely who entered a height into my online, shared record that was several inches shorter than I was...back when I was an 11-year-old child. It's been 25 years since I was that short.

I pressed them that they could at least have asked me how tall I am, or even consulted the previous entries by other doctors in their same system.

There wasn't even an attempt at an excuse. It was plain, simple negligence.

That doctor had also been insisting I needed to let him do an exploratory surgery, despite never even having had me come to the office in person, so I noped right out of there and started telling that story to everyone I thought might consider seeing him.

All of this talk about how "hard" the statistical analysis is, is strange to me. Maybe "advanced" would be a better term? If you get a patient with a contradictory medical history that somehow also contradicts what they are telling you, simply adjust your expected chance of success appropriately (to zero perhaps). In that extreme case, if you get a good outcome, congrats you got lucky. If you don't, it should have 0 impact on how you are evaluated as a doctor.

Institutions face severe penalties for wrongfully sharing patient data, so most opt to just not share any data. Any research that is performed is done internally on local populations with de-identified data sets. A few brave institutions go well out of their way to create and share de-identified data sets publically, but these data sets still undersample the general population. This is a critical problem because certain diseases are highly prevalent in certain regions (e.g., Lyme disease in New England) but unheard of in other regions (e.g., Lyme disease in Colorado). If your ML model is trained on data largely from New England, it's going to diagnose a patient with the classic "target-shaped" rash with Lyme disease even if the patient is from Colorado (high false positive rate). If the model is trained on data from Colorado, it will underdiagnose Lyme disease in patients from New England (high false negative rate). The only way I know to overcome this problem is to create even larger data sets, but this just isn't possible with data privacy laws.

My understanding is that high dimensionality in the domain is exactly where ML excels (so add location as an input), and that is exactly what a medical diagnosis involves. Perhaps the legislation will get there one day.

https://inews.co.uk/opinion/nhs-data-shared-third-parties-we...

Anyway, its out there now.. so watch this space, I guess!

Most big medical co do a lot of data science (Kaiser and others). Very efficient from a managerial pov. Totally useless, medically speaking.

In that I have several CDC funded projects using machine learning in medical analysis and outcome prediction.

Even on extremely well curated data sets this is a fairly hard problem.

Read “The Alignment Problem”, a very good just above pop sci level book about machine learning. They have one example where ML determined seniors with COPD were at reduced risk from pneumonia, and obviously non-sensical result. Patients with COPD wound up in the hospital at a lower rate than average because doctors know they need careful attention right away.

I’m reluctantly pessimistic about humanity’s near-term capability to appropriately weight input from technology as imperfect but inscrutable as current ml.

This is a little like programmers calling the programming language they invented and write in every day garbage. Separate from patient-reported histories, you medical doctors are the ones documenting these histories and hold the decision making power for how it’s done!

Your first sentence would be the equivalent of the inventor of patient history data keeping calling patient history keeping a garbage tool. Which is actually something that would be totally OK to do and say if suffixed with "and unfortunately so far nobody has come up with a better tool and it's not for lack of trying".

I'm assuming you didn't mean "you medical doctors" in the sense it's easy to read in. In any case, what you are doing here is telling one doctor that he is bad at the medical history writing and reading job when in fact he is the one telling you how he is able to spot other doctor's mistakes and trying to correct them. This is like telling one developer that he's bad at his job, that "you developers are the ones writing bad code and hold the decision making power for how it's done" when that developer is actually someone that tries to make things better both through his own maintainably written code (medical histories) and helping others in code reviews to make their code better and not let bad code get into Prod (finding errors in existing medical histories and trying to correct them).

That can be very discouraging, being thrown in with the bad apples. And even good apples can have a bad day or misunderstand something. But I guess you are perfect and have never produced a bug in your life.

There is certainly noise in healthcare data especially when patient-reported, but is it noise to say that a patient having X procedure later does or doesn’t have serious complications? Analyzes of medical care and their consequences can be evaluated and it’s not noise

And big healthcare data has lagged, partially because privacy concerns trump sharing. There are companies selling anonymized medical records for basically every American now though. Big data is coming

Big _bad_ data... Let's see how we fare in 5y, then. My prediction as a clinician with a special interest in stats: close to zero medical progress. But insurance priced by a ML algorithm, and much greater efficiency in coverage and claim denials.

https://www.healthcare.gov/how-plans-set-your-premiums/

The more likely use case for ML is detecting insurance fraud patterns.

I think that's covered under "much greater efficiency in coverage and claim denials."

The codes we do have are mostly CPT4 and ICD-10 for billing purposes. Those are generally pretty accurate, but not detailed enough to reliably assess whether one surgeon is better than another at a particular procedure.

Isn't that already the case, even without formal tracking of outcomes?

To your counter-example, maybe the metrics I described aren’t good enough and should integrate some disease severity criteria or site-weighting or comorbidity score (although as one continues to subdivide the population this way eventually you end up with n=1 and the results are useless again) but like, surely we should be trying to measure something other than the good feels and word-of-mouth of people who have to work with each other?

It physically hurts my brain when I think about how we measure the dumbest shit in software engineering, like which shade of blue to use to improve clickthroughs[0], but when it comes to even attempting quantification of activities which are literally life or death, sorry, too hard, can’t do it. Surgeons will refuse to do hard procedures, insurers will destroy careers, EMRs are full of bad data (so what is the point of the bloody records if they have become that useless‽), surveying patients would cost too much…

I will ultimately defer to the experience of people in the field—I am not a physician or statistician—but sometimes I feel like I’m just being fed arguments repurposed from the bad cops playbook. Oh, we can’t ever possibly start quantifying individual officers’ use of force, because some parts of the city have more crime, and if we do that then those officers will look worse, so they will stop responding to violent calls in those areas, and there’ll be even more crime, so get off our backs man and stop trying to create more objective metrics for accountability.

To be clear I don’t think you are arguing in bad faith and I don’t intend my statement about accountability to suggest that you personally are trying to avoid it or shield bad actors or anything. What you are saying is probably true and I may be wrong to challenge it at all since I have no personal insight into what is going on behind the scenes, and I genuinely appreciate you answering my questions from your perspective and giving me additional perspectives and things to think about. It just feels so, so frustrating as a patient. All I want is some ability to measure risk that’s better than looking up studies on procedure X on pubmed that I’m unqualified to interpret (and which don’t apply anyway because the lead author of the research won’t be doing my procedure), or shaking the magic eight ball.

If I were a physician, I would absolutely want to track the shit out of my own patient outcomes so I could improve, and the amount of resistance that seems to exist (this is not the first time I’ve talked to docs about this and received similar fatalistic answers) is just baffling to me.

We’re not talking about Frogger here, metrics aren’t some high score, if you have an 80% complication rate for some procedure that isn’t necessarily a reflection on you as a practitioner but it would suggest that there is a problem that needs to be identified (bad procedure, bad training, bad support, bad patient, bad luck). Right now, it seems like no one really knows.

This isn’t bullshit alternative medicine, so why, when I scratch beneath the surface, does it so often feel like it is anyway?

[0] https://www.zeldman.com/2009/03/20/41-shades-of-blue/

First, patient privacy laws (while a net good) scare institutions from sharing high quality data. The best you'll get is small batches of de-identified data released infrequently. Patient notes are unlikely to ever be released in large quantities since they can so easily pinpoint some patients.

Second, you need to coordinate thousands of physicians and/or healthcare facilities across the US (or world) to record data on their own performance in a standardized way. Many hospitals do this on some agreed upon metrics (30-day readmission rate, hospital-acquired pneumonia rate, average HbA1c level for a doctor's diabetic patients etc.) largely because they're used to determine government funding/penalties. But at the end of the day, there's no direct incentive for physicians or institutions to collect any other data on their own performance and release it publicly. In fact, there are more risks to doing this than benefits. To solve this problem you need to tie hospital funding with requirements to collect and publicly share performance data while also mitigating punishment.

To physicians' credit, many of us are actually motivated to at least privately collect data on our own performance so that we can improve. But this is incredibly difficult and time consuming—especially for those of us who come into contact with dozens and dozens of patients every day. Sure, better data collection tools would dramatically help us monitor our own metrics, but the only entity with the cash to purchase or create these tools is the hospital, and its reply is going to be, "What's the ROI?" And the answer is honestly probably negative. You may suggest buying/building small relatively inexpensive tools (as I've personally tried), but the hospital isn't interested. Like most large enterprises, hospitals want long-term contracts, dedicated support teams, and tried and true tools. Small tools pose too much of a security risk and maintenance headache.

As a separate observation, any time data is kept, turned into metrics that then become the basis for goals ("I want to have a better-than-average success rate, as a surgeon") then the system gets gamed.

I had a boss once propose to down-rate agile teams that didn't get done everything they took on in a sprint. He apparently didn't realize that teams would immediate game the system by taking on less actual work. They could up their 'point' estimates for each task, and always get the work done.

I'm arguing that a sufficiently comprehensive system would take in to account whatever that doctor realized (and perhaps much more) and compensate for it when determining expected outcomes.

A simple illustration: I have medical issues with no meaningful diagnosis, despite seeing many doctors. If the medical community can't figure out what's wrong, how can they have an understanding of all the relevant factors in determining risk?

This isn't directly related to your point, but why is this even the norm? Why are you going from human to human seeking answers to some arcane mystery like you're on a Skyrim quest.

Imagine a system where you wouldn't need to see any doctors. You would type in all of your symptoms as accurately as you can with as much detail as you can, perhaps with a timeline, and as output you'd get the most likely causes (a diagnosis). Maybe that system would even have your medical history (and that of other people and related diagnoses) to better predictions.

It seems to me that a system like this would be significantly better than going to a couple of humans that are arbitrarily local to you and asking them to figure it out. It is slowly improving all the time just in the form of Google and WebMD. Even before 2010 I accurately diagnosed myself with Bell's palsy with the internet and the doctor begrudgingly asked how I had made that diagnosis. Confirmation from the human doctor was nice but redundant.

I'm also arguing that it's not just a psychological resistance to a data-based evaluation system. That people understand the system would be subject to being gamed, and the overall quality of the work would actually suffer. (A bit analogous to how peer-review and the tenure game has interfered with good science practices.)

But there may be no other doctor.

Think of this way - if you're a doctor specializing in the treatment of septicemia, a lot of your patients will die. If you're really good, you'll likely get the hardest cases. So your "success rate" may be lower than another doctor who isn't as good but doesn't see such tough cases.

You want risk-takers who repeatedly tackle the surgeries and (ideally) get a more positive outcome percentage than a newbie.