Ask HN: I want to scrape a web page...

Re:Ask HN: I want to scrape a web page... by tocomment on Tuesday November 30, 2010 @01:56PM (#1955156)

Just write yourself a Ruby script that downloads a webpage. Then parse the downloaded content with Ruby's equivalent of BeautifulSoup (a forgiving HTML parsing library).

That's pretty much all there is to it. You'll want to manually inspect the HTML from the page you're scraping to find patterns. For example all courses are in <li> tags, etc.

Let me know if you need more help. But I think as a first step write a program to download a web page.

[ Parent | on HN ]

Re:Ask HN: I want to scrape a web page... by bdclimber14 on Tuesday November 30, 2010 @02:08PM (#1955195)

I've never written anything like this... Is there a function like CURL that downloads the page as a string from a URL?

[ Parent | on HN ]

Re:Ask HN: I want to scrape a web page... by chrisa on Tuesday November 30, 2010 @02:47PM (#1955320)

  require 'net/http'
  require 'uri'
  output = Net::HTTP.get(URI.parse("http://www.google.com"))

[ Parent | on HN ]

Re:Ask HN: I want to scrape a web page... by seanmccann on Tuesday November 30, 2010 @03:27PM (#1955461)

curb is a lot faster than net/http

require 'curb' c = Curl::Easy.perform("http://www.google.com) puts c.body_str

[ Parent | on HN ]

Re:Ask HN: I want to scrape a web page... by bdclimber14 on Monday December 06, 2010 @09:46PM (#1977801)

I wanted to thank you for your help. I wrote a script to scrape over 30,000 courses and textbooks from Arizona State University successfully in a couple days.

[ Parent | on HN ]

Re:Ask HN: I want to scrape a web page... by bdclimber14 on Wednesday December 01, 2010 @01:55AM (#1957118)

So apparently accessing a secure URL isn't straightforward... No login is required, its just an HTTPS but it won't go in curb.

[ Parent | on HN ]

Re:Ask HN: I want to scrape a web page... by bdclimber14 on Monday December 06, 2010 @09:45PM (#1977798)

Wasn't an SSL problem but a cookie problem...

[ Parent | on HN ]

Re:Ask HN: I want to scrape a web page... by nc on Tuesday November 30, 2010 @08:04PM (#1956337)

Premature optimization much?

[ Parent | on HN ]