News for hackers. Stuff that matters.
Re:Scrapy Tips from the Pros by IanCal on Wednesday January 20, 2016 @07:33AM (#10937608)
> Or if you want to avoid memory issues for sure, you can send each item to Scala app as it gets collected.

I've done something similar, you can just add a Pipeline (Pipeline Task? not sure of the right terminology) which posts off the data somewhere else. You can also then store the full item logs in S3 so you can go back occasionally and check nothing has been missed. Works an absolute treat.

There are no comments on this comment yet.