News for hackers. Stuff that matters.
Re:Crawl a website with scrapy and store extracted results w... by the_cat_kittles on Monday April 23, 2012 @12:51PM (#3879743)
Have checked out kenneth reitz's requests? Its fantastic, you might like it
Re:Crawl a website with scrapy and store extracted results w... by codehenge on Monday April 23, 2012 @01:40PM (#3879995)
Re:Crawl a website with scrapy and store extracted results w... by the_cat_kittles on Monday April 23, 2012 @03:21PM (#3880492)
thanks, i should have included a code sample too:

    import requests
    from lxml import etree

    jquery_like_page = etree.HTML(requests.get('url').text)