Ask HN: Has Google/Bing won the search engine war?
Recently I've been working on a project that required me to search a limited set of domains.
1)I tried Google Custom Search but they restrict usage to 10000 API calls per day at $50/day. I cannot go higher than this even if I wanted to pay.
2)So I decided to write a crawler that will go to these domains and index their pages so I can put them in ElasticSearch and create a basic search engine.
3)When I created the crawler I came across domains that do not allow crawling their content e.g. Yelp, Craigslist, etc. have locked down their sites for crawling. However I see results from these on Google.
4)It seems the only option is to use their APIs (if they provide one) to get data from these domains. This can be a nightmare to maintain as the number of domains increases.
5)I want to respect the domains' policies and not use shady tactics to crawl their pages.
So essentially these domains allow google and bing to crawl their sites because they are big and established and not having their pages show up on Google or Bing would drastically affect their web traffic but smaller startups would be left out in the cold.
So my question is: What is the possibility of a new search engine emerging if the web is locked down for crawlers?
(
27 comments
| 9 points
| on HN
)
Of course, using this method will not get you data from sites that explicitly ban crawling (like yelp.com).
Disclaimer: I used to work at blekko and I implemented the slashtags feature.
There will never be an all encompassing search engine to rival Google. I would argue, one could not even rival Bing.
The future of search, as with most internet companies moving forward, is all about mastering a niche. You may not be able to beat Google as an overall search engine, but if you dedicate your time and money towards one specific niche, say sports, then you have a fighting chance to beat them at that.
IMHO there are multiple ways to create a search engine: 1)Mastering search within a vertical e.g. sports, music, etc. and then expanding to other verticals. 2)Go after a limited popular area of the web and provide a more personalized search experience(this limits discoverability of less popular sites but there are ways to overcome this) 3)Enhance the results of existing search engines by providing context, personalization, privacy features,etc. (this relies heavily on existing engines which makes it subject to their TOS) 4)Come up with an algorithm that blows Google's or Bing's out of the water and VCs line up to give you money so you can build the infrastructure to make it a reality (I wish I had this too... sadly I don't)
Permitting certain search engines to crawl but not others is anticompetitive and violates the principle of an open web.
> any new search engine will have its roots in "shady" tactics of crawling.
Who cares? If you are successful enough, then you'll negotiate with them later. Don't worry about it now.
Crawl away!
It seems you can contact Yelp and tell them how you plan to use their data and maybe they'll let you crawl their site.
I really want to explore any and all alternative options before I decide to crawl away :)
if they come knocking on your door to take down the content, then worry about it.
You're not going to crawl / index better than Google.
Maybe you could combine the DMOZ directory and Google results somehow.
Rather than replacing Google, try to supplement their services or just somehow provide more value to users.
I typed "sushi" on duckduckgo.com and saw a result from Yelp. If duckduckgo.com used Yelp's API to get that result then you can imagine the complexity of maintaining API connections to hundreds of services like Yelp that do not allow crawling. However if duckduckgo.com crawled/scraped Yelp to get that result then that's against Yelp's TOS which would cause Yelp to send a cease and desist order to duckduckgo.com.
IMHO any crawler that drives traffic back to the original site (like Google, Bing, duckduckgo, etc) should be allowed to crawl the site but Yelp's TOS prevents that.
I used to work at Yelp, and I can tell you that if there were no restrictions on who crawled the site, we'd have been regularly DOSed by robots. I don't (and didn't) like the precedent, but I understand some of the motivation.
If they catch you crawling, they'll send you a cease and desist letter, so do it at your own risk.
Plus, the privacy promotion from them helped drive developers to their sites.
Restrictions on what people can do on the internet is a slippery slope.
http://datamarket.azure.com/dataset/8818F55E-2FE5-4CE3-A617-...
Google Custom Search is the best option in terms of how it allows you to specify the domains you want to search in and save that as a custom search engine and then use the custom search engine's ID in the API request. Furthermore Google Custom Search's result JSON data is more comprehensive than either Bing or Yahoo BOSS.
https://www.google.com/enterprise/search/products/gss.html#p...