News for hackers. Stuff that matters.
Re:A Little Web Spider I Wrote Last Night (code included) by rhett on Thursday March 12, 2009 @05:58PM (#514171)
Here is a web spider in 1 line: perl -MLWP::UserAgent -MHTML::LinkExtor -MURI::URL -lwe '$ua = LWP::UserAgent->new; while (my $link = shift @ARGV) { print STDERR "working on $link";HTML::LinkExtor->new( sub { my ($t, %a) = @_; my @links = map { url($_, $link)->abs() } grep { defined } @a{qw/href img/}; print STDERR "+ $_" foreach @links; push @ARGV, @links} )->parse(do { my $r = $ua->simple_request (HTTP::Request->new("GET", $link)); $r->content_type eq "text/html" ? $r-> content : ""; } ) }' http://www.google.com
Re:A Little Web Spider I Wrote Last Night (code included) by quilby on Thursday March 12, 2009 @08:18PM (#514321)
Re:A Little Web Spider I Wrote Last Night (code included) by yogione on Thursday March 12, 2009 @07:41PM (#514297)
I tested the above script - It works. You are good.
Re:A Little Web Spider I Wrote Last Night (code included) by rhett on Thursday March 12, 2009 @09:09PM (#514366)
thanks, i didn't write it. I remembered that in some magazine from 1999. The link to the author is in the comment below
Re:A Little Web Spider I Wrote Last Night (code included) by jerf on Thursday March 12, 2009 @06:02PM (#514175)
Here's a little web spider I just wrote: "wget -r".

I haven't got a problem with people writing their own quickie scripts, but they aren't really worth putting online without some other compelling reason.

Re:A Little Web Spider I Wrote Last Night (code included) by timf on Thursday March 12, 2009 @05:07PM (#514096)
It's hard to tell if you're after learning or something to use etc., but if you haven't seen it yet, check out http://scrapy.org/ for ideas (or even something to use if that is what you are after).
Re:A Little Web Spider I Wrote Last Night (code included) by a-priori on Thursday March 12, 2009 @10:06PM (#514419)
For completeness, and to keep with the recent meme around here, here's a web spider I wrote a while ago in Erlang.

http://github.com/michaelmelanson/spider