That was my experience as well - training documentation for fresh college grads (i.e. me) directed new engineers to just... send SQL queries to production to learn. There was a process for gaining permissions, there were audit logs, but the only sign-off you needed was your manager, permission lasted 12 months, and the managers just rubber-stamped everyone.
That was ten years ago. Every time I think about it I find myself hoping that things have gotten better and knowing they haven't.
That was also the company where prod pushes happened once a month, over the weekend, and were all hands on deck in case of hiccups. It was an extraordinarily strong lesson in how not to organize software development.
(edit: if what you're really asking is "did every engineer have write access to production", the answer was, I believe, that only managers did, and they were at least not totally careless with it. not, like, actually responsible, no "formal post-mortem for why we had to use break-glass access", but it generally only got brought out to unbreak prod pushes. Still miserable.)
Is it surprising that Engineers in healthcare dont read the actual HIPAA documentation?
Use of health data is permitted so long as it’s for payment, treatment or operations. Disclosures and patient consent are not required.
There are helpful summaries on the US Department of Health and Humans Services website of the various rules (Security, Privacy & Notification)
Source: https://www.hhs.gov/hipaa/for-professionals/privacy/guidance...
This allowance is permitted to covered entities and by extension their vendors (business associates) by HIPAA.
If it wasn’t then, theoretically the US healthcare industry would grind to a halt considering the number of intermediaries for a single transaction.
Example:
Doctor writes script -> EHR -> Pharmacy -> Switch -> Clearinghouse —> PA Processing -> PBM/Plan makes determination
Along this flow there are other possible branches and vendors.
It’s beyond complex.
It just occurred to me that cleaning up our country's data privacy / data ownership mess might have extraordinarily positive second-order effects on our Kafkaesque and criminally expensive healthcare "system".
Maybe making it functionally impossible for there to be hundreds of middlemen between me and my doctor would be a... good thing?
A national identity service, like a normal mature economy, would be a massive public policy and public health win.
But it'd slightly decrease profits. So of course is quite impossible to implement.
Pre-HIPPA, the hospital would tell a news reporter about your status. Now, drug marketing companies know about your prescriptions before your pharmacy does.
In attendance were marketing, biz dev, execs. Ours and theres. But no legal. Hmmm.
And it would have worked if it weren't for those meddling teenagers.
From your comment, I'm inferring they're now allowed to use health records for marketing.
On the would-be due date, a box arrived via FedEx of Enfamil samples and a chirpy “welcome baby” message. Of course there was no baby.
It turns out that using prescription data, admission data and other information that is aggregated as part of subrogation and other processes, you can partially de-anonymize/rebuild / health record and make an inference. Enfamil told me what list they used and I bought it, it contained a bunch of information including my wife’s. I also know everyone in my zip code who had diabetes in 2015.
There’s even more intrusive stuff around opioid surveillance.
> Emfamil samples...
Insult to injury. They're all just ghouls. Profitting from other people's pain.
> what list they used
Who watches the watchers? Nobody.
We had an instance of Lexis/Nexus' (formerly Seisent) demographic database to play with. Most people just can't wrap their heads around how bad things were in the mid-aughts. I was called a "sweaty paranoid kook" by our local paper for simply describing how things worked (no opinion or predictions).
Oh well. Maybe society has started to recognize what's happened.
Thanks for reading my rant. Peace.
Can you please explain more about this or point me towards a source?
Worked on EMRs (during the aughts). Had yearly HIPAA and other security theater training. Not optional for any one in the field.
Of course we had root access. I forget the precise language, but HIPAA exempts "intermediaries" such as ourselves. How else could we build and verify the system(s). And that's why HIPAA is a joke.
Yes, our systems had consent and permissions and audit logs cooked in. So theoretically peeking at PII could be caught. Alas, it was just CYA. None of our customers reviewed their access logs.
--
I worked very hard to figure out how to protect patient (and voter) privacy. Eventually conceded that deanon always beats anon, because of Big Data.
I eventually found the book Translucent Databases. Shows how to design schemas to protect PII for common use cases. Its One Weird Trick is applying proper protection of passwords (salt + hash) to all other PII. Super clever. And obvious once you're clued in.
That's just 1/2 of the solution.
The other 1/2 is policy (non technical). All data about a person is owned by that person. Applying property law to PII transmutes 3rd party retention of PII from an asset to a liability. And once legal, finance, and accounting people get involved, orgs won't hoard PII unless it's absolutely necessary.
(The legal term "privacy" means something like "sovereignty over oneself", not just the folk understanding of "keeping my secrets private.)
please, do tell me, where "estimating better pricing models to extract more profit" fit into those?
> Doctor writes script -> EHR -> Pharmacy -> Switch -> Clearinghouse —> PA Processing -> PBM/Plan makes determination
All the OP mentions happens way after the fact of those paths you described. People have already been charged. Treatment was already decided.
If you want to learn machine learning for healthcare in general, it may help to start problems with tabular data like CSVs instead of images. Image processing is a lot harder, and takes a lot of time and computational power. But it's best to learn what you're interested in the most.
Anyway, first you need to be familiar with basic; Python, machine learning, and popular libraries like scikit-learn, matplotlib, numpy, and pandas. Those are tons of articles, textbooks, and videos to help you learn them.
If you grasp the basics, I think it's better to learn from actual code to train/evaluate models rather than more theories. Kaggle may be a good starting point. They host a lot of competitions for machine learning problems. There are easy competitions for beginners, and competitions and datasets in the medical field.
You can view notebooks (actual code to solve those problems well written by experts) and popular ones are very educational. You can learn a lot by reading those code, understanding concepts and how to use libraries, and modifying some code to see how it changes the result. ChatGPT is also helpful.
If you want to learn image classification, the technology used to detect objects from images and videos is called image classification and object detection. It uses CNN, one of the deep neural networks. You also need to learn basic image processing, how to train a deep neural network, how to evaluate, and libraries like OpenCV/Pillow/PyTorch/TorchVision. There are a lot of image classification competitions in the medical field on Kaggle too[0][1].
To run those notebooks, I recommend Google Colab. Image processing often uses a lot of GPUs, and you may not have GPUs, or even if you have it's difficult to set up the right environment. It's easier to use those dedicated cloud services and it doesn't cost much.
It's hard to learn, but sometimes fun, so enjoy your journey!
[0] https://www.kaggle.com/datasets/paultimothymooney/chest-xray... [1] https://www.kaggle.com/code/arkapravagupta/endoscopy-multicl...
As we grow, I'm ever watchful for our metamorphosis into a big-dumb-company, but no symptoms yet. :)
I'm curious to know the tech stack behind converting unstructured to structured data(for reporting and analysis)
I like this so much I'm definitely stealing it!
The people who have lasted in those roles have built up a large degree of intuition on how their domains work (or they would've done something else).
I am guessing every datascientist that works for a BlueCross BlueShield at individual states deals with processes that touch multiple-million dollars of claims.
We even now have various dueling systems -- one company has a model to tack on more diagnoses to push up the bill, another has a process to identify that upcoding. One company has models to auto accept/deny claims, another has an automatic prior authorization process to try to usurp that later denial stage, etc.
I do not know the chances that any individual provider uses a system to identify what codes to tack onto your bill.
Collaborating with them is great, and has been a great exercise in learning how to explain complex ideas to non-technical business people. Which has the side effect of helping me get better at what I do (because you need a good understanding of a topic to be able to explain it both succinctly and accurately to others). It has also taught me to appreciate the business context and reasoning that can drive decisions about how a business uses or develops data/software.
Collaborating with nontechnical people is oddly my favorite part of doing MLE work right now. It wasn't the case when I did basic web/db stuff. They see me as a magician. I see them as voodoo priests and priestesses. When we get something trained up and forecasting that we both like, it's super fulfilling. I think for both sides.
Most of my modeling is healthcare related. I tease insights out of a monstrous data lake of claims, Rx, doctor notes, vital signs, diagnostic imagery, etc. What is also monstrous is how accessible this information is. HIPAA my left foot.
Since you seemed to be asking about the temporal realities, it's about 3 hours of meetings a week, probably another 3 doing task grooming/preparatory stuff, fixing some ETL problem, or doing a one-off query for the business, the rest is swimming around in the data trying to find a slight edge to forecast something that surprised us for a $million or two using our historical snapshots. It's like playing wheres waldo with math. And the waldo scene ends up being about 50TB or so in size. :D