News for hackers. Stuff that matters.
Re:Crawl a website with scrapy and store extracted results w... by JackC on Monday April 23, 2012 @12:20PM (#3879563)
For really quick one-off scraping, httplib2+lxml+PyQuery is a pretty neat combination:

  import httplib2, lxml, pyquery
  h = httplib2.Http(".cache")
  def get(url):
      resp, content = h.request(  url, headers={'cache-control':'max-age=3600'})
      return pyquery.PyQuery( lxml.etree.HTML(content) )
This gives you a little function that fetches any URL as a jquery-like object:

  pq = get("http://foo.com/bar")
  checkboxes = pq('form input[type=checkbox]')
  nextpage = pq('a.next').attr('href')
And of course all of the requests are cached using whatever cache headers you want, so repeated requests will load instantly as you iterate.

Just something else to throw in the toolbelt ...

Re:Crawl a website with scrapy and store extracted results w... by the_cat_kittles on Monday April 23, 2012 @12:51PM (#3879743)
Have checked out kenneth reitz's requests? Its fantastic, you might like it
Re:Crawl a website with scrapy and store extracted results w... by codehenge on Monday April 23, 2012 @01:40PM (#3879995)
Re:Crawl a website with scrapy and store extracted results w... by the_cat_kittles on Monday April 23, 2012 @03:21PM (#3880492)
thanks, i should have included a code sample too:

    import requests
    from lxml import etree

    jquery_like_page = etree.HTML(requests.get('url').text)
Re:Crawl a website with scrapy and store extracted results w... by jat1 on Monday April 23, 2012 @11:46AM (#3879337)
Also check this out for a pretty good discussion on scraping http://pyvideo.org/video/609/web-scraping-reliably-and-effic...
Re:Crawl a website with scrapy and store extracted results w... by BaltoRouberol on Monday April 23, 2012 @11:59AM (#3879420)
Yeah, I actually learnt scraping from Asheesh :) He's awesome.
Re:Crawl a website with scrapy and store extracted results w... by jat1 on Monday April 23, 2012 @12:30PM (#3879621)
I have been playing with scraping for quite some time now and have my own scripts and stuff, but I found that video informing and there were a few useful snippets I had missed.

Keep meaning to check out more of the Pycon vids

Re:Crawl a website with scrapy and store extracted results w... by danneu on Monday April 23, 2012 @11:02PM (#3882225)
Here's the same functionality written in Ruby using Chris Kite's crawler called Anemone[1]. Gist: https://gist.github.com/2475824. Screenshot: http://i.imgur.com/cbv9A.png

[1]: http://anemone.rubyforge.org/doc/index.html

Re:Crawl a website with scrapy and store extracted results w... by ananthrk on Monday April 23, 2012 @12:14PM (#3879521)
Cool. BTW, is there a reason for naming the file "isullshit_spiders.py" and not as "isbullshit_spiders.py"? :)
Re:Crawl a website with scrapy and store extracted results w... by BaltoRouberol on Monday April 23, 2012 @12:31PM (#3879627)
Oh, that's just a typo. My bad. Edit: there, corrected.
Re:Crawl a website with scrapy and store extracted results w... by hack_edu on Monday April 23, 2012 @10:38AM (#3879003)
I really want to read. Topic is right down my alley.

Unfortunately, the page is literally broken and unreadable on Android ICS with Chrome :(

Re:Crawl a website with scrapy and store extracted results w... by martius on Monday April 23, 2012 @11:35AM (#3879275)
Hi all, thank you for reporting !

This is a known issue and I'm working on it! I'll try to push a "responsive" version today or tomorrow.

Re:Crawl a website with scrapy and store extracted results w... by noinput on Monday April 23, 2012 @11:16AM (#3879195)
Re:Crawl a website with scrapy and store extracted results w... by joshu on Monday April 23, 2012 @09:26PM (#3881961)
Completely unreadable on iPhone too.
Re:Crawl a website with scrapy and store extracted results w... by mumphster on Monday April 23, 2012 @10:50AM (#3879075)
Same using mobile safari
Re:Crawl a website with scrapy and store extracted results w... by jordanmessina on Monday April 23, 2012 @10:52AM (#3879080)
Same using chrome on windows unless I resize the browser and make the width about 1000px
Re:Crawl a website with scrapy and store extracted results w... by martius on Monday April 23, 2012 @11:37AM (#3879289)
Yes, it's true for any viewport with a width of less than 940px. I'll do my best to fix this too.