Ed10101

I did a couple of searches for various terms and you had a huge index for the various terms. Surely such an index is only possible by hooking into search results from another engine, eventually this will lead to some sort of issue?

What's really made it difficult for competitors is being able to match Google's index. Bing crawls a lot of pages but is much more picky about what it index?

Also can you say where you are pulling the results from the Yandex API, Bing, Yahoo or Google itself?

Absolutely love BeautifulSoup, I use it almost everyday in my job. There isn't really a scrapy vs. BS4 divide as you can still use the library with Scrapy, as opposed to it's standard parsing functionality. It also works well with lxml which is considerably faster.

It's also possible to build very performant and large scale crawlers with just BS4 and requests . Though managing the architecture is a bit of a pain, but it definitely can be done.

There are also a number of cases where it's better to use Bs4 than scrapy.

Also using PhantomJS, Selenium and BS4 can provide you with a very powerful data scraping solution.