Our focus has been on capturing patient-reported outcomes - that is the outcome of care as experienced by the patient, measured using objective and validated surveys that are often specific to the condition or treatment. There is now a movement among payers and Medicare to incorporate these kind of patient-centered outcome measures in to reimbursement, although change is admittedly slow.
I'd love to talk to people wanting to make an impact in this field, my email address is francis at outcomes.com
Some aspects are always going to be subjective (eg. impact on quality of life or pain) and IMHO that's OK and we should absolutely attempt to measure them, not least because that information could help inform the treatment itself. Also, by measuring the changes in response over time for a patient, you can attempt to control for individual biases.
I agree that patient satisfaction surveys (in the UK, categorized as patient reported experience measures) can be very prone to bias and while important, are not necessarily correlated with outcomes.
Suppose you assigned people randomly to 101 doctors from 2 populations (A,B). Now suppose A was 10x as likely to die. D(0) get's 0% of A's and 100% B's. D(1) is 1A and 99B. All the way to D(100) that only get's B's.
In that admittedly simplified example you could determine that D(0) did a better job than D(100) by only getting 8x as many deaths even if 9.8x may be statistically irrelevant.
Yes, the real world is vastly more complex. But, while that may make a strict ordering impossible you can likely find out the best doctor is likely in the top quarter and the worst doctor very likely in the bottom quarter, which can be useful.
Picture a score card that said 80% chance in (0% - 20%], 15% chance in the (25%-50%], etc. That's not exactly meaningless information.
Sure, we don't have that information today. But, if we want to collect relevant information we could easily have a subset of doctors with random patient assignment. IMO, that's simply an implementation detail required if we want good information.
PS: US healthcare spending is over 3 trillion per year, rationally even minor improvements are worth large investments.
To me there were two things at play: possibly the sample size of number of patients doctors see with the same problem might be too small and there was genuinely a lot of fear that the doctor’s skill could be boiled down to protocols and statistics.
I think the biggest hurdle toward improving patient outcomes will be alleviating the fear and then after that figuring out how to give better context to the data.
The difference between a 24 student class and a 35 student class is huge in terms of educational outcomes, yet here in the US we tend to cut spending on and underfund education, either by ripping away stable revenue sources like property tax and replacing that with lottery income (which varies wildly from year to year), or by just slowly reducing the amount of money allocated toward schools year after year.
Hence how you end up with places like Seattle having well funded schools due to local ballot measures, yet a similar school in Eastern Washington will get 20% to 30% less funding, and have abysmal results due to it.
Proper education isn't a technology problem, but an investment problem, and rather than investing in schools today, we would rather lock many of those students up in the future here in the US.
Unions exist to lobby for their members; teaching unions represent teachers, not students. All sorts of things might be in the interests of students but contrary to the interests of teachers.
The obvious example is the New York City Department of Education's "reassignment centers". About 600 teachers in NYC are paid their full salary to sit in an office doing nothing. They're not sufficiently trusted to teach due to allegations of misconduct, but neither can they be fired due to strict tenure rules. This system is believed to cost $65m a year. Nobody is happy with the system, but the only way it can meaningfully change is if it becomes easier for the NYC Department of Education to fire teachers.
It's entirely plausible that pupils might be served better by teachers who are brutally over-worked and subject to constant management scrutiny. We know from the private sector that wringing your employees dry is often a very effective way of improving the bottom line. That wouldn't necessarily be the right thing to do, but it simply doesn't follow that what's good for teachers is always what's good for students.
It could meaningfully change if the NYC Department of Education paid their share of arbitration fees, or if the DoE and the union agreed to hire more arbitrators and work through the backlog of cases, or if the DoE got better at reassigning those teachers who have been cleared (many of those in the reassignment centres are teachers who were cleared in their arbitration hearings but the DoE haven't assigned to teaching positions). Partly this is because the DoE doesn't really believe in innocent until proven guilty, and part of that is because they don't trust their own bureaucrats to correctly document genuine misconduct and prove it to the standards the arbitrators require. Fixing that is absolutely on the DoE.
What skilled, dilligent person would want to work in a field that consists mostly of unsupervised interaction with children if one unsupported allegation of misconduct could be a career-ender? That's where you'd be without the arbitration agreement. In much of the rest of the world that's how firing people from any job works - you can't do it without evidence. Somehow those countries manage to be productive. A competent department of education would be able to do the same. Giving an incompetent department more ability to fire people would not be progress.
In the UK I have some knowledge of the work of one of the largest teaching unions the NASUWT.
The members are teachers. They're primarily teachers because they have a passion for teaching. As such their union actively works to improve _learning_ in schools and so lobbies for activity that benefits pupils.
An example. Pupils get a certain amount of money allocated, that money is allocated politically and in a devicive way but does not impact on wages at all, nonetheless the union lobbies for fairer and more transparent distribution of such pupil stipends.
The union does help it's members, but it also does research, shares best practice, and I dare say other things, that help improve _teaching_ too.
They also campaign on social issue - racism, sexism, forced marriage, ... even when it's not directly related to teaching.
Perhaps your characterisation is true of specific teaching unions in USA?
The issue with the Reassignment centres appears to be they can't sack teachers for doing unlawful things without a trial (seems reasonable, presumption of innocence), they lack evidence to go to trial, but they feel the teacher can't be trusted.
What's your solution? Sack people without due process? So sack teachers when there's a suggestion they might have done something wrong? In situations where pupils can easily make false allegations (without repercussions on themselves) that enables pupils to get teachers fired very easily; is that fair in your opinion?
And what do teachers want? Educate students to the best of their ability. Yes, I'm giving the benefit of the doubt here that many teachers actually want to do the educate and ain't just in it to "get rich" because if one's sole motivation is making money then god knows there are a myriad of better choices than becoming a teacher.
Imho this representation of unions as "oh so greedy money grubbers who only do it for the money" really irks me, especially considering that the original reason for unions actually being a thing was to counter exactly such "greed money grubbing" ways by exploiting labor in mass when it was still living in serfdom or straight up slavery.
Unions do not just serve the purpose of "making union members richer", even if that has become the de-facto reality in some places. Without unions it would be doubtful anything like workplace safety regulations would ever have become a thing, many standards for healthy and productive working, we take today for granted, are the direct result of unionization in some way or another.
To get back on topic, teachers are opposed to objective measures because if you can objectively tell if a teacher is bad, the union will lose influence over pay and employment security, which is where their power is anchored. Thus, they will of course promote studies that show the failures of such objective measures and try to discredit those that show the opposite. Just like any other interest organisation.
That doesn't mean that I have a problem with the existence of interest organisations, but let's not pretend they are something they're not.
Or that since there exists no such measurement pushing for one is dishonest and betrays other agendas.
In medicine, I don't think there's a clear long-term metric (lifespan? financial success? happiness?), and there's at least two nasty selection bias that people generally only see a doctor when (self-report) something is wrong, and that doctors specialise. But let's say you solve those problems: What kinds of questions[1] are you going to ask that are supposed to predict satisfaction with the doctor?
[1]: https://londoncalling.co/2013/02/simple-customer-feedback-id...
I agree with you that the more abundant data (and reduced selection bias) available for schools makes measurements easier for them than for medicine, but I don't think that it's any easier to define an appropriate metric for judging them.
I think a better metric would take into account overall health history, years paid into Social Security (as a proxy for income), and food/housing stability. Look at those three factors, and you can get a grip on how a person's life has gone in a way you can compare to others.
In cases where the same teacher was subject to multiple assessments (there's a Math Nerd writing for Bloomberg who presented this, I think at TED), there was no correlation between the multiple measurements.
Straight-up noise.
Cathy O'Neil, "MathBabe", "Algorithms are Opinions in Code".
Starting at 2m22s here: https://www.youtube.com/watch?v=_2u_eHHzRto
Doctors have effectively been able to defend their "turf," from hostile encroachment, while teachers have not, not because the situations do not contain substantial parallels, but rather because doctors are politically strong while teachers are politically weak.
While with teachers it's kinda the opposite; Even tho their whole job is to know and teach things, many people have a way easier time disagreeing with them straight out of principle.
Wonder how much of that boils down to socialization aka in what contexts children are introduced to these professions?
Its much easier to punch down than it is to punch up!
As far as I know no-one has come up with a metric that's actually useful for students and teachers.
1) "Each patient is unique" just like "each student is unique"...but we still give standardized tests (to the grumbling of below-average teachers and administrators)
2) If the sample size is too small because the disease is rare, there may be a genuine measurement problem. If the doctor or hospital's sample size is too small but the illness is common, then they should be referring those patients to providers/facilities with the requisite experience levels.
But then, perhaps I would be a "below-average" teacher, and my opinion might therefore similarly be discarded as worthless. We'll never know for certain, because I don't make a habit of boarding sinking ships - especially when I'm slated to receive blame for their sinking after boarding them. (And I did at one point very much want to enter the field of secondary school education.)
(Slight modification: I won't board a sinking ship and take blame for it's sinking after the fact without some substantial advantage, such as excellent remuneration, being offered as well - teaching offers no such advantage, except perhaps self-actualization, and I can neither eat nor sleep in that.)
(I find it curious that another poster takes the opposite tack, claiming that teaching exists only to maximize the financial success of students, whereas medicine has a more nuanced end. I confess to finding this position truly bizarre.)
For example, I have a simple model that ends up showing that an ICU doing everything it's supposed to do (patient isolation, high hand hygiene compliance, perfect diagnosis, etc.) can have a four-fold difference in MRSA infections over a year by chance alone.
Data are also often tremendously thin -- I've seen profiling efforts over tens of millions of patient-lives in which a given provider may only have a handful of records. That's not enough to draw strong inferences, and year-to-year variance is going to be tremendous.
It also turns out that there are very strong clusterings of patterns around facilities: a senior physician or surgeon can (and does) drive practice, quality, and methods for much of the rest of the medical staff.
The single best predictor of quality I recall was volume of procedures. Practice (and standardisation) help tremendously.
You can iterate and fix your mistakes.
My experience with the world of medicine is that it is way less scientific and evidence based. This is because it can be more complex, but it does lead to doctors being able to ignore the fact that their treatments don't work.
Bad doctors can bury their mistakes and blame other factors, whereas bad engineers quickly get found out.
I think the thing we really need is a better patient experience model that maximizes trust and information transfer and helps ensure compliance. (Disclaimer: I led a team building one of the top patient experience platforms for several years)
Perhaps any doctor would have better stats if they got to have the same pool of patients as those who remained with Warwick, without having other less motivated patients to drag them down.
We can't know if this is the case until we collect the data on patients who left Warwick's care.
We don't really need truely objective assessments. As an analogy, imagine we randomly added or subtracted 3 minutes to the finishing times of all the runners in the Boston marathon. We wouldn't have a truely objective assessment of each runner, but it's still a useful guide about who is likely to beat whom in the next marathon.
> - Each patient's case is unique and cases with the highest levels of difficulty are often treated by the most experienced people. These cases, of course, are likely to have worse outcomes than simple cases which may be treated by less experienced (worse?) physicians.
This problem can be alleviated to a large degree by risk-adjusting the outcomes: For each patient, estimate the most likely outcome and compare the actual outcome to the estimate.
> - Clinical outcomes are largely recorded by the same people treating the patient so reported outcomes are often erroneous or frankly fraudulent.
I don't see this as an insurmountable obstacle. For example, you could have an independent body randomly sample the reported outcomes to check that they are accurate, and apply some kind of penalty if they are not.
I don't think the analogy fits. There's no single metric (like runner speed/time) to evaluate competence.
> This problem can be alleviated to a large degree by risk-adjusting the outcomes: For each patient, estimate the most likely outcome and compare the actual outcome to the estimate.
Agree.
> I don't see this as an insurmountable obstacle. For example, you could have an independent body randomly sample the reported outcomes to check that they are accurate, and apply some kind of penalty if they are not.
That would be a good solution. Part of the problem is finding a truely independent body. Medicine is over-run with various governance and regulatory bodies (several of which have been shown to be little more than rent collectors). And again, there's the problem of deference to eminence combined with small in-bred communities within each speciality or sub-specialty (who would likely be the only people with the training to evaluate their peers reliably).
Let me suggest such a metric: A T-value. That is, the degree to which the actual outcome deviates from the expected outcome.
For example, say you have a hospital that does stem cell transplants. For each patient before the treatment you assess the "chance that the patient will die within 1 year" based on that patient's age, sex, BMI, heart and lung function, type of disease, time since last relapse, quality of donor match, etc. From this you estimate that 19.6% of the hospital's patients who are treated over a particular period will die within 1 year, with a standard deviation of 3.7%. The actual mortality rate for this group of patients turns out to be 26.2%. So the T-value is +1.78; this is the 'single metric' used to evaluate the competence of the hospital.
> there's the problem of deference to eminence combined with small in-bred communities within each speciality or sub-specialty (who would likely be the only people with the training to evaluate their peers reliably).
Keep in mind that expertise is only required in estimating pre-treatment "chance that the patient will die within 1 year". Determining whether a patient is alive after a year and calculating the T-value can be done by anyone. And even when estimating expected mortality, there is plenty of evidence that a simple algorithm can actually beat the experts - Daniel Kahneman devotes a whole chapter to this point in 'Thinking Fast and Slow' [1].
[1] Chapter 21: https://www.amazon.com/dp/B00555X8OA/ref=dp-kindle-redirect?...
If I were to publish my ratings, one doctor will be unfairly penalized. However, hundreds of patients will benefit by being able to switch to a better doctor. In this hypothetical situation, do the rights of one doctor outweigh the rights of so many patients?
We also have record levels of advertising by not only drugmakers but also doctors and hospitals. If there is a shortage of providers, why is so much being spent on demand generation efforts?
If twice as many people who are going to one doctor or hospital are ending up dead or crippled, and there are no discernible confounding factors (age, socioeconomic, co-morbidities, etc), then I want to know. One might even say the patient has a right to know. The burden should be on the provider/facility to convince the patient to trust them nonetheless.
If we go with your proposal then the inevitable outcome is that the best providers (particularly surgeons) will engage in metrics arbitrage by refusing to treat patients whose co-morbidities and complications aren't adequately captured by standardized coding systems and clinical guidelines. Is that really the outcome you want?
As for putting the burden on providers to convince patients to trust them, good luck with that. We currently have shortages of providers in many areas and specialties due to price fixing and supply constraints. So most patients have to take what they can get regardless of trust.
Based on the doctors I know, the medical world does not have an especially profound understanding of how to deal with large amounts of data. Their specialities are diagnosis or specialist surgery, not data.
If the data is so scary we have to hide it then there are glaring problems that need to be addressed. Sure there are misleading edge cases; but believing that accurate data will be worse than word of mouth is, quite frankly, unwarrented.
If a provider's or facility's (or entire speciality's) numbers look bad, that's important information for the consumer to possess. Hiding the numbers should not be an option.
Or do you think that car crash test results, and airplane crash data, and health department inspection findings should also be kept secret?
More problematically, if you can't articulate a quantifiable standard that will indicate a provider's quality level, then then entire concept seems ill-defined. I.e., when you refer to the "best providers", what does that even mean?
Their statement only seem bizarre when you view it from the perspective of individualism. Another way to look at it is - let me (ab)use Star Trek quote: "The needs of the many outweigh the needs of the few". You seem to make absolute statements that in fact are relative and rooted in one ideology you chose to follow.
> If a provider's or facility's (or entire speciality's) numbers look bad, that's important information for the consumer to possess. Hiding the numbers should not be an option.
Forgive me the harshness - I believe this is a simplistic way of looking at the problem. OP clearly showed systemic forces at play, and those are important to consider. You're saying "hiding the numbers", as if they were perfect numbers hidden away in a safe. Problem is there are not, and even when they might be very nuanced. A point raised by OP: "best providers (particularly surgeons) will engage in metrics arbitrage by refusing to treat patients" - I think it is a perfectly reasonable threat, which you chose to ignore.
> Or do you think that car crash test results, and airplane crash data, and health department inspection findings should also be kept secret?
Again going back to OP's example - doctor's might be incentivised not to treat patients which make their numbers look bad. None of your examples share similarity in this sense (and possibly in many more).
If your concern is with metrics arbitrage, then it is not necessary to use quantitative metrics to satisfy concerns about bodily autonomy. One might instead require providers to furnish anonymized records of all the adverse events their patients have experienced, along with any mitigating factors they think absolve them of responsibility.
I think the issue of quantification is actually a red herring. A simpler example will indicate whether or not there are differences in our ethical intuition. I am scheduled for a "routine" surgery, but yesterday's patient who was having the same surgery by the same doctor had a major artery sliced, bled out and died. All the staff are aware of what happened, but no one tells me. In fact, even if I ask, people are instructed to say nothing. If I possessed this information, I would almost certainly decide not to proceed. I go ahead with the procedure based on the understanding that death is a much more remote possibility than those treating me happen to believe. Is this or is this not a violation of my bodily autonomy?
If it puts out of business doctors and facilities (perhaps even specialties?) whose patients agree to care based on grossly inaccurate understandings of their track record, then it would more than pay for itself.
You have hit upon the core problem though. Outside of a few limited areas where we have clear evidence-based medicine guidelines there is no reliable quantifiable standard for measuring provider quality.
Old school surgeons will insist that they can reliably make the diagnosis without imaging (and that's probably where you get a 20% false positive from). Some institutions will push this notion too - either to reduce imaging expenditure or increase surgical turnover (sad but true).
That opens a pile of other issues. For reference, when I showed up at the ER it was a Canadian hospital. They called the surgeon in early, for me. I think word went out that they were spinning up an OR early and the ultrasound tech rushed in. So cost and turnover weren't really an issue. Everyone was salary, the equipment owned, and nobody standing to profit by cutting into me without need. Frankly, everyone had a good laugh at how quickly the system came together. But I did fall into that category of people who showing up on evenings weekends that, statistically, don't do as well.
Its extremely hard to definitively define best practises. You'd be surprised how many practitioners still argue for what look like clearly outdated ways of doing things. There's also the argument that in some people's hands an outdated way of doing things is safer than a newer (better?) way of doing things in their hands. eg. A surgeon who has been practising for 30 years but only recently learned to use a laparoscope.
I doubt ambulance chasers do anything other than muddy the waters and enrich themselves at the healthcare systems expense.
This is the same for software, and is equally annoying (especially as a self-taught Dev), but physicians at least have the assessment of people who have had to endure intellectually difficult course work. You also have several assessments of direct physician ability (boards) and possibly other things.
As for software, I can basically follow tutorials, watch some videos, maybe do a few small projects, and then be able to convince someone to actually give me a job. That's my only measurement -- whether someone will pay for my skills.
You guys are much closer to objectively verifying someone's ability, so hopefully that doesn't seem as vexing for you.
Every individual is unique and yet we are able to segment folks using multiple variables and draw predictions for a bunch of things across businesses - it's just like saying "every shopper online is unique", which is true, yet we still find ways to influence them of target them in somewhat meaningful ways.
You should be able to do the same with patients, by clustering them in larger groups that make sense (for cancer many trials refer to specific mutations nowadays to define whether a patient is expected to get efficacy from a specific drug).
The only question in the end is whether or not you have enough sample size for each doctor. Maybe not. Then it would make sense to at least draw comparisons between groups of specialists across multiple hospitals, at least.
Even if metrics are imperfect to begin with, there is no excuse not to start at least, and improve over time.
This is relevant as I think the views here help determine where the focus in healthcare ought be.
Oh my, I've been suffering for back/neck pain for years. I will randomly get flare-up (usually when it gets cold or when I do sport) and things will get worse and worse. I've been going to different PT/phisio/osteopath for the last 10 years and nothing never changed. My last doctor told me an X-ray wouldn't do much and sent me to another PT...
Now I'm considering visiting a chiropractor eventhough reddit has told me not to trust them... But I mostly just endure the pain every day.
I understand the pain, keep trying and try whatever's best for you.
I will tell you that yoga/pilates/other non-ballistic, moderate resistance exercise that targets stabiliser muscles helps many people.
I would avoid surgery at all costs unless you have cord compression, a focal lesion or a fracture.
I'd be very wary of chiropractors - I've seen several patients with vertebral artery dissections (with incapacitating brain stem strokes!) because of neck manipulation. Of course, I have a selection bias but even a small risk for this seams like too much.
I have following Chris Centeno, M.D. for some years. As far as I know he tries all/some approaches including stem cells, chiropractors, exercises etc.
If you do end you being treated by them, let me know the outcome - good or bad.
It will require formalizing the informal process you describe (whereby "cases with the highest levels of difficulty are often treated by the most experienced people"), most likely by creating additional certification levels.
> Reported outcomes are often erroneous or frankly fraudulent
Could you elaborate?
- History and physical exam are extremely subjective.
- Medical notes and records can be massaged to sound better than reality (The patient didn't have a stroke after their operation. They had "minor lower limb weakness which we anticipate will improve over the next few weeks") or attributed to all sorts of things (patient's cognitive function hasn't worsened, they have some "fatigue" following surgery).
- People can be very selective about which studies are ordered.
- Junior and ancillary staff are reluctant to report poor outcomes about established senior people.
I'll give a real-life example: Professor X is a "world-renowned expert" on neurosurgery at an academic mecca. He claims a "1-2%" complication rate on aneurysm surgeries in the clinic and at any conferences. His residents know he performs outdated surgeries with spectacularly bad outcomes but no one would be willing to talk about this openly unless they want to torpedo their career prospects.
Someone close to me, a very experienced nurse, reported some misconduct on the part of a senior physician. This was both required by law and an ethical duty.
She essentially got blackballed from that hospital as a result.
And before you judge anyone too harshly, talk to me again when you have 300k of debt and you're a fourth year resident stuck in a singularly focused career pathway.
Also, the reality is most people have so much else going on to concern themselves with that these things are given little thought.
Accumulation of power inevitably leads to abuses of said power, at least in my experience.
When cure confirmed by patient, diagnosis is proven correct, data is captured, reputation increased among participants in consensus that were correct.
You could even use this as human-in-the-loop machine learning model training for open models.
If you want to explore what it might take to build this, we've just launched an alpha of our data/ml network that can facilitate a dapp built on top of Synapse https://synapse.ai/
Happy to reach out and chat more.
I'm still not sure I understand what you mean. Could you give an example?
> When cure confirmed by patient, diagnosis is proven correct, data is captured, reputation increased among participants in consensus that were correct.
It's not that simple. Most patients are not either 'cured' or 'not cured'. There are multiple possible outcomes at each stage of diagnosis and management.
> You could even use this as human-in-the-loop machine learning model training for open models.
Doubtful. Current machine learning algorithms are painfully inept when given clinical data (outside of very limited use cases). I'd love to be proven wrong though.
> If you want to explore what it might take to build this, we've just launched an alpha of our data/ml network that can facilitate a dapp built on top of Synapse https://synapse.ai/ Happy to reach out and chat more.
Sure, email?
More likely things are this way because the relevant people and institutions don't want their performance measured, and they have the power to keep things this way.
The problem is comparing it in a useful way across different institutions.
"Comparing it in a useful way across different institutions" seems like a BIG deal!
Institutions revealed to be inferior would have to improve close down. Those proven to do great work would be richly rewarded, but at all levels there'd be a new level of scrutiny.
This would be of huge benefit for patients, save tons of lives etc, but also put a lot of strain on doctors and hospitals, and I can see how people with power in those fields don't want this to happen and thus don't make it happen.
It's also entirely possible I babble too much about things I know nothing about here. If so, I'd appreciate learning about it!
I would really appreciate insight on how this could be achieved.
There are several issues which are particularly vexing:
- The distinct lack of verifiable, objective markers of physician competence.
- Each patient's case is unique and cases with the highest levels of difficulty are often treated by the most experienced people. These cases, of course, are likely to have worse outcomes than simple cases which may be treated by less experienced (worse?) physicians.
- Clinical outcomes are largely recorded by the same people treating the patient so reported outcomes are often erroneous or frankly fraudulent.
- This is made worse by the hierarchical nature of clinical medicine and deference to seniority and title.
- Medicine is parochial so clinical practises for the same disorder vary tremendously. You might be treated a dozen different ways for the same disorder and presentation depending on the facility and especially on the specialty that ends up treating you.
- Outcomes are not necessarily determined by clinician ability. There are several other factors at play: the pre- and post-care (such as work-up by ancillary staff or ICU care after a surgery), the cohesiveness of the facility and its efficiencies (or lack thereof), availability and preferences for resources such as medical devices, drugs and hospital equipment which may be largely out of the hands of the physician.