If you are not going to learn AWK, R, Python or Julia the next best thing is Open Refine.
> OpenRefine (formerly Google Refine) is a powerful tool for working with messy data: cleaning it; transforming it from one format into another; extending it with web services; and linking it to databases like Freebase.
Please note that since October 2nd, 2012, Google is not actively supporting this project, which has now been rebranded to OpenRefine. Project development, documentation and promotion is now fully supported by volunteers. Find out more about the history of OpenRefine and how you can help the community.
Re:Using wget and grep to explore inconveniently organized f...
by conductor on Friday October 02, 2015 @03:59PM
(#10320914)
This reminds me of Manning's case.
"That Manning was convicted of computer fraud seems to suggest that using wget on a U.S. government computer to download large numbers of files can be considered the digital equivalent of trespassing – even if it's on turf you're otherwise allowed to access."
Re:Using wget and grep to explore inconveniently organized f...
by morninj on Friday October 02, 2015 @06:41PM
(#10321759)
Well, sort of. In the Manning case, the government argued that wget was unauthorized on government-owned client machines. Their argument in that case wouldn't apply to the public's use of wget to send requests to government-owned servers.
Re:Using wget and grep to explore inconveniently organized f...
by samstave on Friday October 02, 2015 @07:25PM
(#10321915)
Then we need a scan of all government machines to see if there was a precedent of it being installed across many thousands of machines to suggest that this was not an actual individual breach of policy. and show that the government was remiss in its ability to enforce the policy at all.
Re:Using wget and grep to explore inconveniently organized f...
by engi_nerd on Friday October 02, 2015 @11:21PM
(#10322460)
Government information security and technology people, in my experience, are not necessarily forgiving or understanding in any way. And I mean "understanding" in both senses of that word. They neither care to understand your needs nor necessarily understand much about technology. I see so much extra work done just because the security person in charge of writing a procedure had no understanding of what the phrase "volatile memory" means. And that's just one instance from my work; I wish I could share more but it all makes me sad.
All this is a very long way of saying that some of these people may not have even understood what wget is used for, or believed it was unauthorized for use even though it is a standard tool.
Re:Using wget and grep to explore inconveniently organized f...
by baldfat on Friday October 02, 2015 @02:51PM
(#10320477)
The URL is nerds??? I look at the page and there was nothing on there that seems to show it as a acronym either. Just the title of the blog. Not sure I like that.
Re:Using wget and grep to explore inconveniently organized f...
by csense on Friday October 02, 2015 @05:26PM
(#10321453)
One interesting thing the author mentions -- interesting despite being somewhat irrelevant to the topic -- is the astonishing fact that the vast majority of people don't know how to use Ctrl+F.
Re:Using wget and grep to explore inconveniently organized f...
by ape4 on Friday October 02, 2015 @03:07PM
(#10320565)
Its so sad there isn't a site with everything in plain sensible formats. data.gov seems like a good start. But a plain text UI (like gopher or ftp) seems better. A subfolder for each state. And a subfolder for each department. Maybe something like Linux's /proc for laws and federal data.
Re:Using wget and grep to explore inconveniently organized f...
by danso on Friday October 02, 2015 @02:52PM
(#10320481)
Enigma is a great service and they might have a list of the Section 333 exemptions but even with the sophistication of their discovery algorithms, I'd be amazed if they had this as structured data...mostly because right now, it exists officially as a list of PDFs with some basic information, e.g. operator name and stated mission. Extracting location of operator probably wouldn't be too hard...detect if PDF follows standard letter format and then run address extraction algorithm on the first (or 5) last lines...
Obviously, someone from enigma.io could give some manual direction (such as deciding that city/state are actually important data points to extract), but it probably wouldn't be as detailed as dataset as what The Verge and Bard College have manually curated [1]
If you are not going to learn AWK, R, Python or Julia the next best thing is Open Refine.
> OpenRefine (formerly Google Refine) is a powerful tool for working with messy data: cleaning it; transforming it from one format into another; extending it with web services; and linking it to databases like Freebase. Please note that since October 2nd, 2012, Google is not actively supporting this project, which has now been rebranded to OpenRefine. Project development, documentation and promotion is now fully supported by volunteers. Find out more about the history of OpenRefine and how you can help the community.
"That Manning was convicted of computer fraud seems to suggest that using wget on a U.S. government computer to download large numbers of files can be considered the digital equivalent of trespassing – even if it's on turf you're otherwise allowed to access."
http://www.washingtonpost.com/news/worldviews/wp/2013/07/30/...
All this is a very long way of saying that some of these people may not have even understood what wget is used for, or believed it was unauthorized for use even though it is a standard tool.
https://www.propublica.org/nerds
https://www.propublica.org/nerds/item/doc-dollars-guides-col...
Nerd has a negative connotation as opposed to Geek. I know seems weird but getting called nerd was very negative before Geek/Nerd was cool.
Definition: 1. a stupid, irritating, ineffectual, or unattractive person. 2. an intelligent but single-minded person obsessed with a nonsocial hobby or pursuit: a computer nerd. [1] http://dictionary.reference.com/browse/nerd?r=75&src=ref&ch=...
I submitted it here: https://news.ycombinator.com/item?id=10321439
Obviously, someone from enigma.io could give some manual direction (such as deciding that city/state are actually important data points to extract), but it probably wouldn't be as detailed as dataset as what The Verge and Bard College have manually curated [1]
[1] https://github.com/voxmedia/data-projects/tree/master/verge-...
-nolist disable the link list feature in dumps.
See also -crawl and -traversal