News for hackers. Stuff that matters.
Re:Snowden Used Low-Cost Tool to Best N.S.A by vwinsyee on Saturday February 08, 2014 @06:09PM (#7203277)
Contrary to the current HN title, the article points out:

Evidence presented during Private Manning’s court-martial for his role as the source for large archives of military and diplomatic files given to WikiLeaks revealed that he had used a program called “wget” to download the batches of files. That program automates the retrieval of large numbers of files, but it is considered less powerful than the tool Mr. Snowden used.

So the tool wasn't wget. curl, perhaps?

Re:Snowden Used Low-Cost Tool to Best N.S.A by 3pt14159 on Saturday February 08, 2014 @07:26PM (#7203582)
Having done this type of work before for a legitimate purpose, it is almost certainly a python or perl script with a nice library in front of it that makes it easy to follow links.

wget is too brittle, not extensible enough, and not as maintainable as a nice python script.

Re:Snowden Used Low-Cost Tool to Best N.S.A by jrochkind1 on Thursday February 13, 2014 @09:45AM (#7231611)
I believe Manning actually used Windows batch scripting to automate wget, or so the government alleged from forensics at the trial. (I observed a couple days of the trial).

Manning did not have the tech skills of Snowden though, she wasn't neccesarily doing things in the most effective or elegant ways, but it worked.

Re:Snowden Used Low-Cost Tool to Best N.S.A by emmelaich on Sunday February 09, 2014 @06:51AM (#7205229)
Probably; but could be something like lftp. It's name belies its capabilities.

Or maybe Kermit? Half-smiley; only if he's a masochist. http://www.kermitproject.org/ckscripts.html

Re:Snowden Used Low-Cost Tool to Best N.S.A by wslh on Saturday February 08, 2014 @10:38PM (#7204238)
Wget is also single threading which is a slow strategy to download pages.
Re:Snowden Used Low-Cost Tool to Best N.S.A by Fasebook on Saturday February 08, 2014 @10:53PM (#7204286)
that's what xargs -n x is for
Re:Snowden Used Low-Cost Tool to Best N.S.A by wslh on Tuesday February 11, 2014 @08:45PM (#7221944)
Can you elaborate?
Re:Snowden Used Low-Cost Tool to Best N.S.A by sebastianavina on Saturday February 08, 2014 @07:51PM (#7203660)
the other day I had the task to batch download product pictures from a website... every picture had a sessionid on the uri so I could't make a simple image wget. I wrote a simple python script that wrote a shell script with a lot of "wget -E -H -k -p \n sleep 30" and ran it trough a cloud server for a couple days... after that, some simple scripts for renaming the pictures, some regular expressions here and the, and voila, 250k perfectly named pictures for my product catalog... (it's for an intranet, so I guess I wont have copyright problems"
Re:Snowden Used Low-Cost Tool to Best N.S.A by eli on Sunday February 09, 2014 @02:30AM (#7204754)
FYI, you have exactly the same copyright issues on an intranet. You're just less likely to get caught, I guess.
Re:Snowden Used Low-Cost Tool to Best N.S.A by kudu on Saturday February 08, 2014 @06:19PM (#7203312)
curl is just a library with a slim command-line interface. It can't scrape pages by itself. Perhaps you're thinking of curlmirror? Even then, I doubt it can be considered more powerful than a good wget configuration.
Re:Snowden Used Low-Cost Tool to Best N.S.A by kurtsiegfried on Sunday February 09, 2014 @05:39AM (#7205098)
Nutch/Solr could provide a way to do a crawl, refine parameters, and then feed into a tool to download the actual resources.