Using wget and grep to explore inconveniently organized federal data

Re:Using wget and grep to explore inconveniently organized f... by spectre256 on Friday October 02, 2015 @02:06PM (#10320193)

This feels like a data set that might already be on the amazing https://enigma.io

[ Parent | on HN ]

Re:Using wget and grep to explore inconveniently organized f... by otoburb on Friday October 02, 2015 @02:38PM (#10320401)

Enigma's secure URL isn't working. The working address is http://enigma.io

[ Parent | on HN ]

Re:Using wget and grep to explore inconveniently organized f... by spectre256 on Friday October 02, 2015 @03:24PM (#10320662)

That's disappointing. I actually didn't even check that they used HTTPS, I figured it was safe to assume they do. Guess not.

[ Parent | on HN ]

Re:Using wget and grep to explore inconveniently organized f... by danso on Friday October 02, 2015 @02:52PM (#10320481)

Enigma is a great service and they might have a list of the Section 333 exemptions but even with the sophistication of their discovery algorithms, I'd be amazed if they had this as structured data...mostly because right now, it exists officially as a list of PDFs with some basic information, e.g. operator name and stated mission. Extracting location of operator probably wouldn't be too hard...detect if PDF follows standard letter format and then run address extraction algorithm on the first (or 5) last lines...

Obviously, someone from enigma.io could give some manual direction (such as deciding that city/state are actually important data points to extract), but it probably wouldn't be as detailed as dataset as what The Verge and Bard College have manually curated [1]

[1] https://github.com/voxmedia/data-projects/tree/master/verge-...

[ Parent | on HN ]