I agree. Private enterprise has built some incredible data set. Google Street View is an incredible public good, and who knows what would be built on top of it if it truly were public, rather than owned by Google.

But some things can't really be built by private enterprise. The government can compel data reporting. You can pay credit card companies to sell you their data, but only the government should have the power tell every employer "you must tell us the names and salaries of all your employees". (Credit agencies essentially do this too, but that means they're big enough to require government regulation.)

It can be onerous, though, and sometimes unnecessarily so. When I was at a charter school, every here I had to compile a data set on the demographics of our students to ensure we weren't mistreating people based on race or something. It's a very important thing for the government to be checking on, but given that we already had to send all our student's demographics, attendance, and discipline accounts to the state government, it felt redundant.

I feel like the government has been moderately good at making these data sets more accessible, I'd love to see more resources put into that, as well as making their collection smoother. Building effective, efficient systems is incredibly difficult, and requires constant investment to respond to changes.

I harp on "infrastructure" a lot, and I'm coming to realize that what I mean is less "roads, bridges, and trains" and more "efficient, effective systems of all types".

Don't forget that private companies like Google very often create value not by collecting data as such, but by crawling through 50 different US state datasets and untold thousands of county/town datasets to figure out property lines, ZIP code maps, address numbers, etc. There's a symbiotic relationship here, much like research in the natural sciences: basic research and basic data collection is a public good, but there's plenty of value to be added by private industry. It's a win-win as long as the people at large are able to recapture some of the value created by private industry (rather than it being concentrated in the hands of a few stakeholders).

Doesn’t the article state that lots of data the government collected was just incorrect in obvious ways? 29% of facilities ProPublica contacted said the EPA’s data on them was wrong.

And why didn’t the government pay for an interface and analysis that the public could easily use?

I think people believe their tax money could be spent more wisely and competently.

As I read it, the data about emissions specifically is self-reported by facilities:

> We reached out to each of the top 200 facilities (ranked by the level of nearby cancer risk) to ask them if their emissions reporting was accurate — and if not, whether they would resubmit 2014-18 data to the EPA. Of the 109 companies that responded to us, 71% confirmed that their reported emissions were correct, and 29% noted errors, which we asked them to correct.

I live in the western hemisphere's largest integrated industrial complex (Freeport, TX integrated with the eastern edge of Houston as well). Note that Freeport, TX has ZERO state or federal EPA VOC analyzers which can actually detect which chemical is leaking. They can only detect "this amount of something with either sulfur or N-O bonds -- no clue what though!". Completely fucking useless for an area which manufactures something like 15-20% of all USA domestic chemicals.

The entire east side of Houston metropolitan area has only 3 air quality monitors which test for these kinds of chemicals. During huge major events like the ITC fire, they often show no increased pollution at all. I lived next to leaks every day and because I worked in the plants I knew the smells - one day acrylates, next day thiols, next day hydrocarbons, etc. But the 3 monitoring sites over 10 miles from me showed nothing at all.

Here is the one "correct" monitoring station near the chemical plants of Houston: https://tceq.maps.arcgis.com/apps/webappviewer/index.html?id... but several of its analyzers are often offline/broken/pending maintenance. Here's a map of all the other ones: https://tceq.maps.arcgis.com/apps/webappviewer/index.html?id... Generally single/dual color dots mark "not-useful" monitoring sites which might measure only PM2.5 or Ozone, for example. The 4+ color dots are generally useful.

The data used by ProPublica is far worse than the data here -- because what ProPublica used was "self-reported" data from the chemical plants. But living next to them and working in them, I know that many leaks are never reported and many leaks are never even known internally! Our government's data collection is a travesty. ProPublica couldn't use the real air quality measurements because having 2-3 points across 1000 mi^2 is completely useless for the wind models they wanted to apply to the problem.

We don't actually have any data. The government is failing us. They need to spend about $1 million per air monitoring station and build them along the perimeters of each plant so that leaks can be assigned to the offending companies, and they need to be built near housing so that we know how families are being affected.

ITC fire which blanketed houston's sky in smoke: https://abc13.com/deer-park-fire-2019-itc-houston-air-qualit...

The companies submitted bad data, and it was not until ProPublica visualized what the data were saying that it became obvious.

If anything it shows the government is not providing enough oversight on the data provided.

> 29% of facilities ProPublica contacted said the EPA’s data on them was wrong.

I don't think it's necessary to point out why this is not very convincing

> And why didn’t the government pay for an interface and analysis that the public could easily use?

Because "the government" -specifically, the executive branch- has its upper leadership torn out and replaced with a completely new bunch of people with hugely different ideology if not an explicit political interest in dismantling the work of their forebears every 4 or 8 years. This is most obvious with the recent Trump admin, but applies in degree to every single administration.

There used to be an internal, cross-disciplinary office that did API and visualization work for accountability reasons (unfortunately I forget the name). They did great work. It was dismantled and 90% of it was not replaced. The 10% that was replaced consists of (1) internal work (EIA being a good example, DOT being a bad one) that is driven entirely by the good will of employees who are fighting tooth and nail for any amount of budget to expose any amount of the work they do. And (2) external work by any number of massively wasteful contracting programs, like SBIR, where I used to work.

Reagan's SBIR program mandates a certain amount of money to be spent on tiny (100k-1m) contracts which are either useless or unused, because there's no way to actually convert anything to a long-term status. Most of that money goes to defense, but make no mistake that the work there is also completely useless. The whole program should be lit on fire.

> I think people believe their tax money could be spent more wisely and competently.

It can, if people stop voting for republicans who intentionally destroy the only functional programs and then divert the money to completely worthless contractors in the name of "free markets".

> Because "the government" -specifically, the executive branch- has its upper leadership torn out and replaced with a completely new bunch of people with hugely different ideology if not an explicit political interest in dismantling the work of their forebears every 4 or 8 years.

And what's worse is that one political party has been running a decades-long campaign to literally make government worse in order to get people on board with dismantling big sections of it. It's not even a conspiracy theory, it's an actual part of their public campaign platforms that people deliberately vote for.

It's hardly only the republicans blowing government funds on useless programs, Obama's team did the same with bogus 'clean coal' funding funnelled into public-private partnerships with coal outfits (to the tune of ~$8 billion via the DOE), gave loans to the renewable energy companies with the worst products on the market via insider connections (Solyndra), etc. It's an across-the-board problem.

If we really wanted to do large-scale infrastructure projects that were successful, like nationwide high-speed-rail, maybe we should study how China does it.

Conversely 70% of the data is good and valuable.

As a counterpoint, in my hometown there's a grassroots effort to collect and monitor the air quality because no one, including the feds, will. The data can be appreciated without dogmatism.

My personal bias here would be to say, "lowering the cost of technology, civic organization (whatever form that might take), empowering the individual, and removing barriers to entry wins again."

> This is a clear case of the value that governments can provide with your tax money, and something that almost certainly would not be provided (free of charge!) by the free market.

Pretty sure an HOA could fairly easily collect this data (I have a setup for various chemicals in my back yard). I have shared said data with my HOA to change the treatment plan for common areas. If an HOA did that they can sue a plant for any issues and alter emissions.

Government (big g) is nothing but an organized group of citizens agreeing to certain rules. There’s no reason any random organization can’t gather information, sue, etc. exactly the same as government.

We’re raised to think government is necessary, but I’m apart of many organizations that effectively function as self-governance and collective bargaining. At the same time they don’t need forced taxes paying for bombs in other countries either…

All it takes is a minor amount of organization. A great example of this is the HAM radio community, farming co-ops, churches, etc.

> Pretty sure an HOA could fairly easily collect this data

I agree, but on a large scale this would mean that every HOA needs to have access to someone who can/is willing to do this. If you expand this to other environmental factors (e.g. noise pollution, waterway health, etc) I could see that becoming a large burden on the few people in each community that care enough to collect environmental data. Seems more efficient to pay someone to do it full-time for a larger group.

Sure an HOA can easily hire someone, a building manager can and often does buy a service to monitor building detectors. That’s kind of my point, we already have these scenarios where people not government solve problems.

> I could see that becoming a large burden on the few people in each community that care enough to collect environmental data.

What you’re saying is not enough people care. So the people that do care are taking money from people who don’t care to get what they want. That’s called stealing when the government isn’t doing it…

That’s my point from above, we’re taught to need government. We don’t. Never did.

> we’re taught to need government

That's an uncharitable way to cast GP's comment, and not really true. We do understand by now that the state is just a better solution than private entities relying on "market forces" for certain desired outcomes.

Back to the original point: the problem is scalability. If everyone goes through the same process, running into the same roadblocks and suffering the same pitfalls, independently and disconnectedly, that is by definition wasted effort, ie, unproductive for the economy. If we want such things to succeed, ie, be productive and contribute to the commonwealth, we need to share knowledge, which requires some level of centralization at this time (decentralized knowledge management techniques are still in their infancy and IMO require architectural changes in our telecommunications infrastructure to properly support).

> We do understand by now that the state is just a better solution than private entities relying on "market forces" for certain desired outcomes.

I’d say the opposite is true actually. I don’t understand how any mandated entity is better. I’ve formed plenty of organizations, governments aren’t necessary. It’s supposed to help mediate force. Instead people use the governments monopoly on force to get what they want.

In terms of scale, we have planned parenthood, Churches, habitat for humanity, and so many others that work at scale. If pollution was an issue as described people would care. HOAs would hire organizations to monitor pollution and then HOAs would sue for damages. Almost exactly what the EPA does btw. Except the government gets the money and people in the country fund pollution research like this.

> I’ve formed plenty of organizations

Have you run a country? Do you know how that goes? No? Ok then.

> In terms of scale, we have planned parenthood, Churches, habitat for humanity, and so many others that work at scale.

Would you like to know how much each of these groups get in tax writeoffs, subsidies, and aid from the government?

> Almost exactly what the EPA does btw.

A comment about how we don't need government, and the argument is that government is already doing the job. Amazing.

> I have a setup for various chemicals in my back yard

Could you share any more information about this?

Totally, our $30,000,000,000,000 in government debt got us some measurement data that is barely a passing grade (~1/3 incorrect data). They kind of dropped the ball after that but a private company was able to clean the data and make it neat visualization that got up votes on HN.

My faith in the government is restored.

Wow yeah, you nailed it! Government debt is directly related to programs like this, and this was probably the lion's share of the expenditure!