News for hackers. Stuff that matters.
Re:Crawl a website with scrapy and store extracted results w... by JackC on Monday April 23, 2012 @12:20PM (#3879563)
For really quick one-off scraping, httplib2+lxml+PyQuery is a pretty neat combination:

  import httplib2, lxml, pyquery
  h = httplib2.Http(".cache")
  def get(url):
      resp, content = h.request(  url, headers={'cache-control':'max-age=3600'})
      return pyquery.PyQuery( lxml.etree.HTML(content) )
This gives you a little function that fetches any URL as a jquery-like object:

  pq = get("http://foo.com/bar")
  checkboxes = pq('form input[type=checkbox]')
  nextpage = pq('a.next').attr('href')
And of course all of the requests are cached using whatever cache headers you want, so repeated requests will load instantly as you iterate.

Just something else to throw in the toolbelt ...

Re:Crawl a website with scrapy and store extracted results w... by the_cat_kittles on Monday April 23, 2012 @12:51PM (#3879743)
Have checked out kenneth reitz's requests? Its fantastic, you might like it
Re:Crawl a website with scrapy and store extracted results w... by codehenge on Monday April 23, 2012 @01:40PM (#3879995)
Re:Crawl a website with scrapy and store extracted results w... by the_cat_kittles on Monday April 23, 2012 @03:21PM (#3880492)
thanks, i should have included a code sample too:

    import requests
    from lxml import etree

    jquery_like_page = etree.HTML(requests.get('url').text)