News for hackers. Stuff that matters.
Re:Show HN: Explore Doctors' Disciplinary History by jamesfly on Monday October 09, 2023 @08:14AM (#37819624)
Thank you. This data should be available in all states, but each state has a proprietary site I have to write a new scraper for and write a new data pipeline for. I'm thinking about open-sourcing the project and gathering community support for the rest now that I have a good pattern set.

Yes, I was harmed by a doctor and the medical board dismissed my complaint. This made me curious about what the board actually does take action for, but this wasn't searchable because all that was available were PDF scans that weren't digitized/OCR'd. I also noticed some documents that were referenced in doctors' profiles had disappeared. As part of this, I archived every PDF available on the wayback machine to protect against that going forward.

I also want to give credit to a similar site that I took inspiration from called 4patientsafety.org, which did some of the same stuff I have here, but completely manually. I chatted with the creator of that and he agreed making an automated version would help because they fell behind updating it due to how intensive the manual process was.

Re:Show HN: Explore Doctors' Disciplinary History by cobertos on Monday October 09, 2023 @09:35AM (#37820327)
Is there a link to the community where this is being developed? I was unable to find one on the site itself. I'd be interested in writing a scraper for my state and the digitization pipeline.
Re:Show HN: Explore Doctors' Disciplinary History by the_sleaze9 on Monday October 09, 2023 @10:19AM (#37820700)
I'd be interested in helping out as well, looking forward to it
Re:Show HN: Explore Doctors' Disciplinary History by jamesfly on Monday October 09, 2023 @08:18PM (#37827093)
Thank you. I'm going to refine the site and data a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
Re:Show HN: Explore Doctors' Disciplinary History by jamesfly on Monday October 09, 2023 @08:17PM (#37827090)
Thank you for offering to help. I'm going to refine the site and data a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
Re:Show HN: Explore Doctors' Disciplinary History by heliostatic on Monday October 09, 2023 @09:36AM (#37820328)
You may want to look at courtlistener.org as inspiration--similar set of problems, but in a different space obviously.
Re:Show HN: Explore Doctors' Disciplinary History by jamesfly on Monday October 09, 2023 @07:20PM (#37826686)
Thanks but the site seems to be down. Do you have another link?
Re:Show HN: Explore Doctors' Disciplinary History by levelhorizon on Tuesday October 10, 2023 @04:13AM (#37829645)
It's www.courtlistener.com
Re:Show HN: Explore Doctors' Disciplinary History by jamesfly on Thursday October 19, 2023 @12:34AM (#37938458)
Thanks! I'm looking at the Contribute page and seeing some cool ideas.
Re:Show HN: Explore Doctors' Disciplinary History by yorwba on Monday October 09, 2023 @10:14AM (#37820655)
> I archived every PDF available on the wayback machine

How did you tell archive.org to store the PDFs? It doesn't seem to have worked in all cases, since when I went to check out the single "order vacating automatic cancellation of certificate", there was no snapshot yet.

https://web.archive.org/web/20231009140931/https://www2.mbc....

Re:Show HN: Explore Doctors' Disciplinary History by jamesfly on Monday October 09, 2023 @07:18PM (#37826674)
I have to admit I missed a few hundred out of about 18k PDFs. I didn't keep track of which ones I archived on my initial pass, so I need to rerun the whole script which takes about a week with Web Archive's rate limit, so I've just been putting it off. I was also thinking people who visit them can click the "archive" button in cases it's not yet archived to crowdsource the effort, like you seem to have done.
Re:Show HN: Explore Doctors' Disciplinary History by codeTired on Monday October 09, 2023 @01:42PM (#37823068)
I’m interested in helping out.

I have done a lot of research in medicine due to my poor experiences with even top specialists. Ignoring symptoms, denying side effects of meds I took, even though those were present on the label.

Do you have a discord?

Any plans on allowing people to comment on physicians?

Any plans of adding how much doctors have received from corporate interests?(as this data is available)

Re:Show HN: Explore Doctors' Disciplinary History by jamesfly on Monday October 09, 2023 @08:16PM (#37827081)
Sorry to hear you've also had poor experiences. I too had to delve into Pubmed to figure out what was going on with me after the doctor harmed me.

Thank you for offering to help out. I do want to refine the site and data a little more over the next few days and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo.

I've now made a discord here https://discord.gg/bXdk4CuR7B for anyone interested in contributing.

Comments on doctors should already be enabled via Disqus if you scroll to the bottom of a profile page.

Yes, I definitely want to add that data!

Re:Show HN: Explore Doctors' Disciplinary History by johnmaguire on Monday October 09, 2023 @02:03PM (#37823268)
I'm also interested in writing a scraper for my site (Michigan) if time allows.
Re:Show HN: Explore Doctors' Disciplinary History by jamesfly on Monday October 09, 2023 @08:19PM (#37827099)
Thank you. I'm going to refine it a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
Re:Show HN: Explore Doctors' Disciplinary History by noman-land on Monday October 09, 2023 @02:49PM (#37823808)
Please open source this. I would love to help.
Re:Show HN: Explore Doctors' Disciplinary History by jamesfly on Monday October 09, 2023 @08:18PM (#37827097)
Thank you. I'm going to refine the site and data a little more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B for anyone interested in contributing.
Re:Show HN: Explore Doctors' Disciplinary History by mywrathacademia on Monday October 09, 2023 @07:17PM (#37826668)
Open sourcing this would be a great idea and a lot of people would happily contribute, it already looks great. The biggest secret is that most doctors do more harm than good when a patient has anything worse than a cold. The psychopaths who couldn't make it as CEOs or serial killers end up becoming doctors, in charge of people's health.
Re:Show HN: Explore Doctors' Disciplinary History by jamesfly on Monday October 09, 2023 @08:20PM (#37827110)
Thank you. I'm going to refine the site a bit more over the next week and maybe do a second state myself to make sure my pattern generalizes well before opening up the repo, but I've now made a discord https://discord.gg/bXdk4CuR7B for anyone interested in contributing.