Re:U.S. OMB to release largest index of government data in t...
by danso on Tuesday February 24, 2015 @01:04PM
(#9102342)
Besides the usual quality/depth issues of any dataset, this will likely be a constant obstacle (emphasis mine):
> These indexes, if developed and maintained properly by the agencies, will reveal a vast trove of government information to the public.
However, even just having a straight-up list of what exists is a really good start...a lot of data (and information in general) is not really kept secret, it's just obscure.
A great data anecdote comes to mind, from an investigation that found Florida police officers with severe misconduct charges were continually employed. The records of their misconduct were public record, in a SQL database, but reporters didn't previously know about it, and the agency who kept such data...well, it's not their job to monitor it:
> The misconduct database was the big thing. It had multiple tables in it. Then there was a separate employee database, which was a state-wide database. It’s basically like a glorified rolodex. (The state) is in charge of certification, which is why they keep the database. There is a form that officers have to fill out if they change jobs or departments. It is the cleanest set of data I have ever worked with. There was no big clean up with the data. Sometimes you get a data set and find out it has errors or wrong information. Everywhere we turned this data pointed us correctly.
> This was a case where the government had this wonderful, informative dataset and they weren’t using it at all except to compile the information. I remember talking to one person at an office and saying: “How could you guys not know some of this? In five minutes of (SQL) queries you know everything about these officers?” They basically said it wasn't their job. That left a huge opportunity for us.
At least with an index of datasets, you can grep it for something like "misconduct" or "inspection" or "investigation" and start from there.
Re:U.S. OMB to release largest index of government data in t...
by esonderegger on Tuesday February 24, 2015 @04:23PM
(#9103626)
I'll remain skeptical about OMB's ability to know about all the datasets from federal agencies, because as you said, there is a wealth of government data that is "public" but obscure.
Another example is the Department of Defense's budget justification data. The PDF documents that go into the annual President's budget are made public and are easily found on agency websites. They look like this:
What the public would have no way of knowing, though, is that attached to those documents are files with the extension of *.zzz. Those are really zip files, but need to be renamed because DoD computer systems tend to view anything with a zip extension as toxic. Inside those zip files are the xml files used to create the PDF document. The data is both clean and comprehensive.
I'm writing a set of scrapers that will download the publicly available PDF files and extract the XML, so if someone wanted to import them into their own database for querying, they could. I really wish that weren't necessary, though.
Edit: I should mention that those xml attachments only exist for the Procurement and RDT&E budgets, which account for about a third of total DoD spending.
Re:U.S. OMB to release largest index of government data in t...
by Shivetya on Tuesday February 24, 2015 @07:20PM
(#9104503)
If anything perhaps people can show them how to organize their data better to see where all the duplicated efforts are and reign in some of the spending.
Then maybe if the right data is available we can query all the regulations to find obsolete or weed out all the special interest driven ones
Re:U.S. OMB to release largest index of government data in t...
by rrggrr on Tuesday February 24, 2015 @01:32PM
(#9102544)
For those wondering when China, the EU or others will overtake the United States' reserve currency status and/or safe-haven status consider this data dump.
Is the USGOV transparent? No. Is it more transparent than any alternative? Yes. Auditing the Fed and Ft. Knox would be welcome additions. FOIA access to US and State Courts would be nice as well.
Re:U.S. OMB to release largest index of government data in t...
by afiedler on Tuesday February 24, 2015 @03:25PM
(#9103255)
The Fed also releases all of the memos that are used by the FOMC to set monetary policy. It just releases them with a five year lag. It also releases transcripts of all FOMC meetings and conference calls, plus all presentation slides used [1].
It really is quite transparent, and I'd argue that releasing things with a codified lag is a good balance between the need for secrecy and the public's right to know.
You can even see the options for wording of the FOMC's statements, considered to be some of the most sensitive material at the Fed. It's contained in the section of the "Bluebook" produced prior to the FOMC meeting called "Alternative Statement Language". [2]
Source: used to work there and make some of the charts and models that go into those documents.
Re:U.S. OMB to release largest index of government data in t...
by snowwrestler on Tuesday February 24, 2015 @03:30PM
(#9103286)
Yes, the "audit the Fed" movement is really about increasing the influence of Congress on Fed policy decisions, not financial controls the way we normally think of them.
The Fed was designed to be independent of Congress and the fight over whether that was the right decision is still being waged.
Re:U.S. OMB to release largest index of government data in t...
by sswaner on Tuesday February 24, 2015 @09:52PM
(#9105007)
And rrgrr's concern about China overtaking the US will happen shortly after the Fed looses it's independence. The open deliberations of Congress lend little to the fight against special interests subverting common sense.
Re:U.S. OMB to release largest index of government data in t...
by taylorwc on Tuesday February 24, 2015 @02:42PM
(#9102995)
I believe certain of its actions are subject to audit by the Office of the Inspector General and others by the OMB. Regardless, your point stands that a comprehensive audit of the Fed's actions is not readily available.
> These indexes, if developed and maintained properly by the agencies, will reveal a vast trove of government information to the public.
However, even just having a straight-up list of what exists is a really good start...a lot of data (and information in general) is not really kept secret, it's just obscure.
A great data anecdote comes to mind, from an investigation that found Florida police officers with severe misconduct charges were continually employed. The records of their misconduct were public record, in a SQL database, but reporters didn't previously know about it, and the agency who kept such data...well, it's not their job to monitor it:
http://ire.org/blog/on-the-road/2011/12/20/behind-story-trac...
> The misconduct database was the big thing. It had multiple tables in it. Then there was a separate employee database, which was a state-wide database. It’s basically like a glorified rolodex. (The state) is in charge of certification, which is why they keep the database. There is a form that officers have to fill out if they change jobs or departments. It is the cleanest set of data I have ever worked with. There was no big clean up with the data. Sometimes you get a data set and find out it has errors or wrong information. Everywhere we turned this data pointed us correctly.
> This was a case where the government had this wonderful, informative dataset and they weren’t using it at all except to compile the information. I remember talking to one person at an office and saying: “How could you guys not know some of this? In five minutes of (SQL) queries you know everything about these officers?” They basically said it wasn't their job. That left a huge opportunity for us.
At least with an index of datasets, you can grep it for something like "misconduct" or "inspection" or "investigation" and start from there.
Another example is the Department of Defense's budget justification data. The PDF documents that go into the annual President's budget are made public and are easily found on agency websites. They look like this:
http://www.saffm.hq.af.mil/shared/media/document/AFD-150130-...
What the public would have no way of knowing, though, is that attached to those documents are files with the extension of *.zzz. Those are really zip files, but need to be renamed because DoD computer systems tend to view anything with a zip extension as toxic. Inside those zip files are the xml files used to create the PDF document. The data is both clean and comprehensive.
I'm writing a set of scrapers that will download the publicly available PDF files and extract the XML, so if someone wanted to import them into their own database for querying, they could. I really wish that weren't necessary, though.
Edit: I should mention that those xml attachments only exist for the Procurement and RDT&E budgets, which account for about a third of total DoD spending.
- https://project-open-data.cio.gov/
- http://www.digitalgov.gov/resources/how-to-get-your-open-dat...
And you can see how agencies are currently performing at http://labs.data.gov/dashboard/offices
Can you point to some examples or links for the .zzz files you mentioned?
Then maybe if the right data is available we can query all the regulations to find obsolete or weed out all the special interest driven ones
Currency <-- confidence <-- accountability <-- transparency.
Is the USGOV transparent? No. Is it more transparent than any alternative? Yes. Auditing the Fed and Ft. Knox would be welcome additions. FOIA access to US and State Courts would be nice as well.
[1] http://www.federalreserve.gov/publications/annual-report/201...
It really is quite transparent, and I'd argue that releasing things with a codified lag is a good balance between the need for secrecy and the public's right to know.
You can even see the options for wording of the FOMC's statements, considered to be some of the most sensitive material at the Fed. It's contained in the section of the "Bluebook" produced prior to the FOMC meeting called "Alternative Statement Language". [2]
Source: used to work there and make some of the charts and models that go into those documents.
[1]: http://www.federalreserve.gov/monetarypolicy/fomchistorical2...
[2]: http://www.federalreserve.gov/monetarypolicy/files/FOMC20080..., page 27
The Fed was designed to be independent of Congress and the fight over whether that was the right decision is still being waged.
I honestly can not picture what a policy audit of the fed would even look like. What's the deliverable?