Re:Scrapy Tips from the Pros
by IanCal on Wednesday January 20, 2016 @07:33AM
(#10937608)
> Or if you want to avoid memory issues for sure, you can send each item to Scala app as it gets collected.
I've done something similar, you can just add a Pipeline (Pipeline Task? not sure of the right terminology) which posts off the data somewhere else. You can also then store the full item logs in S3 so you can go back occasionally and check nothing has been missed. Works an absolute treat.
I've done something similar, you can just add a Pipeline (Pipeline Task? not sure of the right terminology) which posts off the data somewhere else. You can also then store the full item logs in S3 so you can go back occasionally and check nothing has been missed. Works an absolute treat.