News for hackers. Stuff that matters.

A comment on Ask HN: I want to scrape a web page.... Parent comment

Re:Ask HN: I want to scrape a web page... by chrisa on Tuesday November 30, 2010 @02:47PM (#1955320) 

  require 'net/http'
  require 'uri'
  output = Net::HTTP.get(URI.parse("http://www.google.com"))
[ Parent | on HN ]
Re:Ask HN: I want to scrape a web page... by seanmccann on Tuesday November 30, 2010 @03:27PM (#1955461)
curb is a lot faster than net/http

require 'curb' c = Curl::Easy.perform("http://www.google.com) puts c.body_str

[ Parent | on HN ]
Re:Ask HN: I want to scrape a web page... by bdclimber14 on Monday December 06, 2010 @09:46PM (#1977801)
I wanted to thank you for your help. I wrote a script to scrape over 30,000 courses and textbooks from Arizona State University successfully in a couple days.
[ Parent | on HN ]
Re:Ask HN: I want to scrape a web page... by bdclimber14 on Wednesday December 01, 2010 @01:55AM (#1957118)
So apparently accessing a secure URL isn't straightforward... No login is required, its just an HTTPS but it won't go in curb.
[ Parent | on HN ]
Re:Ask HN: I want to scrape a web page... by bdclimber14 on Monday December 06, 2010 @09:45PM (#1977798)
Wasn't an SSL problem but a cookie problem...
[ Parent | on HN ]
Re:Ask HN: I want to scrape a web page... by nc on Tuesday November 30, 2010 @08:04PM (#1956337)
Premature optimization much?
[ Parent | on HN ]