Case mix and case severity are real concerns. Physicians (and hospitals) vary tremendously in each, and often the best physicians get the hardest cases, which, if not accounted for in metrics, make them look bad.

Data are also often tremendously thin -- I've seen profiling efforts over tens of millions of patient-lives in which a given provider may only have a handful of records. That's not enough to draw strong inferences, and year-to-year variance is going to be tremendous.

It also turns out that there are very strong clusterings of patterns around facilities: a senior physician or surgeon can (and does) drive practice, quality, and methods for much of the rest of the medical staff.

The single best predictor of quality I recall was volume of procedures. Practice (and standardisation) help tremendously.