2. You can run them locally. We're not running the scrapers anywhere, or storing extractions anywhere.
3. This is a big, big question. Right now, the answer is dependent on the use case. Rather than trying to make the world's biggest database, we're going to respond to community needs and build this kind of thing as it comes up.
4. https://measuresforjustice.org/ is doing something like this! We're interested in creating incentives for police departments to make their data more accessible and transparent.
What I would expect to see is something like:
1. Here's what data we want from each police department each day. Here's what value you should use to indicate that data is not available.
2. Here's a list of police departments. Write a scraper. If it passes tests to show it's generating valid data, and code review, we will run the scraper in a daily basis storing the output in this database.
3. Here's how you can query our database.
We've only had paid staff for about 6 weeks. We've run several experiments and started from scratch a few times over the years. We've been slowly inching the project forward in our spare time; I was the only volunteer for much of that time, and I can't even code!
1. Sounds great, we're building something like this.
2. Even making a list of police departments is a big challenge. We've made a good start but have work to do.
3. Yep, soon we'll have something to query.
I do have a few questions too:
1. Will this scale? One problem with scrapers is that they break when people update their website. I'm imagining this problem multiplied by 18,000 and compounded by each scraper potentially being written by a different volunteer.
2. Where are the scrapers getting run?
3. How do the documents that the scrapers collect get transformed into usable data?
4. It seems to me like a scalable solution would be a standard to report data, a law to compel police departments to follow that standard, and then a system to collect that data and make it available. Do you work with police departments at all on data reporting?