News for hackers. Stuff that matters.
Re:Scrapy: New Python web crawling & scraping framework (bui... by agentbleu on Sunday December 28, 2008 @01:27PM (#412074)
is anyone from scrapy here, i have some tech questions? is there an irc group?
Re:Scrapy: New Python web crawling & scraping framework (bui... by lowkey on Monday December 29, 2008 @08:24AM (#412980)
I am looking for some community action for scrapy. It looks useful for a project I'm working on currently using BeautifulSoup but not digging the sluggish performance.

I am having trouble resolving the docs to the code. Is there an IRC, mailing list or forum?

Re:Scrapy: New Python web crawling & scraping framework (bui... by agentbleu on Monday December 29, 2008 @01:51PM (#413291)
the irc group on freenode has some groupies there who help. I have it installed, dont bother with debian, use a clean install of ubuntu

here are some notes to get started from a clean install (replace your own vars)

install Ubuntu 810 apt-get update apt-get install subversion adduser --home /home/bleu bleu su bleu svn co http://svn.scrapy.org/scrapy mv scrapy-trunk scrapy ls scrapy branches tags trunk sudo root as root apt-get install python-twisted apt-get install nano su bleu source ~/.bashrc pico ~/.bashrc #add this to end of file: export PYTHONPATH=/home/bleu/scrapy/trunk

python >>> import scrapy quit() scrapy/trunk/scrapy/bin/scrapy-admin.py startproject myproject