News for hackers. Stuff that matters.
Re:Who exactly is crawling my site? by nostromo on Friday October 18, 2013 @02:24PM (#6573079)
Crawlers and spambots are the scourge of medium to small websites.

I run a small wiki that gets just a few thousand human hits a day. But according to the server logs 90% of server hits are crawlers and spambots, so I'm using 10 times the resources I really need to serve customers.

I finally resorted to blocking entire data centers and companies that crawl constantly but send no traffic.

I feel like search engines should crawl websites in proportion to the traffic they send. For example, Yandex, Baidu, and Bing were all crawling my website hundreds or thousands of times a day, but never sending a single visitor (or in the case of Bing sending single-digit visitors). It's an absurd waste of resources, so I blocked them completely.

Re:Who exactly is crawling my site? by victorf on Friday October 18, 2013 @03:41PM (#6573578)
Add a time limit to robots.txt and block anyone who violates it. Maybe also add a nonexistent honeypot (I disallow /wpadmin on my static website haha). I've been fail2banning a lot of shitbirds and my traffic logs before and after are quite different.
Re:Who exactly is crawling my site? by greglindahl on Friday October 18, 2013 @10:39PM (#6575121)
Fortunately for us new(er) search engines (I work at blekko), only a small fraction of websites actively block crawlers from new search engines. It's a chicken and egg problem: we're going to have a big index for a long time before we have big traffic.

I also have a bunch of hobby websites, and I agree that bing crawls a lot more than Google and sends me a lot less traffic. And Yandex and Baidu have very little English-speaking usage, so they aren't sending me much traffic.

Re:Who exactly is crawling my site? by PaulHoule on Friday October 18, 2013 @07:08PM (#6574551)
It's actually a lot worse if your site is large since crawlers will make even more requests crawling a large site.

If I see that a crawler is sending me any traffic at all, I will accept that, but if the amount of traffic is zero, I put them in robots.txt and in the IP block list. Although I try hard to make sure my images are all clean creative common images, I block the robots that are there to find copyrighted images because these ones exist to give me nothing but trouble.

Re:Who exactly is crawling my site? by gabemart on Saturday October 19, 2013 @10:19AM (#6576487)
>I block the robots that are there to find copyrighted images because these ones exist to give me nothing but trouble.

I've never heard of these bots before. Where did you hear about them, and how do you block them?

Re:Who exactly is crawling my site? by leephillips on Saturday October 19, 2013 @08:48AM (#6576211)
Is there a list of good IPs to block, similar to the hosts files that people pass around?

EDIT: chrsstrm mentions something below

Re:Who exactly is crawling my site? by ryandrake on Friday October 18, 2013 @03:03PM (#6573330)
Chicken and egg. Should they 1. crawl you first, or 2. send you traffic first? If 1, how long until they should either send traffic or stop, and if 2, how?
Re:Who exactly is crawling my site? by pavel_lishin on Friday October 18, 2013 @03:19PM (#6573432)
Why not adjust the crawl rate based on how often the site's content changes? If DuckDuckGoogle crawls me once an hour for a week, and notices that I only updated my content twice, why not scale the crawling back to a more reasonable rate, like once a day?
Re:Who exactly is crawling my site? by lostlogin on Friday October 18, 2013 @05:12PM (#6574045)
This seems logical, however what about the person who updates rarely (once or twice a month) but get a lot of notice when they do. Search engines wouldn't want to miss the opportunity to get that traffic. I wonder if there is some sort of reverse process you could opt into - ie never crawl me until I ask to be crawled. Disclaimer: I know little about this sort of thing.
Re:Who exactly is crawling my site? by gizmo686 on Friday October 18, 2013 @05:40PM (#6574193)
You could probably do this by having a robots.txt file that blocks all crawling. When you want to be re-crawled, you can edit it to allow the relevant spiders to crawl you. I would imagine that spiders do not automatically re-check sites which disallow everything often, but Google (and probably others) let you manually submit URLs to be crawled.
Re:Who exactly is crawling my site? by walshemj on Friday October 18, 2013 @07:13PM (#6574577)
But a disallow all robots.txt will kill your sites traffic - its not going to work like you think it is.
Re:Who exactly is crawling my site? by pavel_lishin on Friday October 18, 2013 @07:17PM (#6574584)
Presumably you could set a minimum interval - an hour, or a day. This would make sure that big updates are noticed, without putting undue stress on the server.
Re:Who exactly is crawling my site? by toomuchtodo on Friday October 18, 2013 @03:11PM (#6573380)
Could they not run webhooks where my site can call to them when to do a full or partial crawl? PubSub for web crawling, if you will.
Re:Who exactly is crawling my site? by ceejayoz on Friday October 18, 2013 @03:17PM (#6573412)
Sitemaps (http://www.sitemaps.org/) help a bit in that regard, as crawlers can check the sitemap and only crawl updated content.
Re:Who exactly is crawling my site? by unhappyhippie on Friday October 18, 2013 @05:21PM (#6574088)
Thanks.
Re:Who exactly is crawling my site? by sergiosgc on Saturday October 19, 2013 @03:58AM (#6575726)
PUsH: http://en.wikipedia.org/wiki/PubSubHubbub

Google is the only crawler I see using the protocol on my site. It does make Google updates occur within minutes, so that alone is reason to implement push.

Re:Who exactly is crawling my site? by leephillips on Saturday October 19, 2013 @08:58AM (#6576234)
I thought this was only being used for RSS - interesting!
Re:Who exactly is crawling my site? by guard-of-terra on Friday October 18, 2013 @05:47PM (#6574235)
Do you/did you receive traffic from DDG? Because DDG uses Yandex for [some] organic search results and if you block its spider you will also partially block DDG.
Re:Who exactly is crawling my site? by nostromo on Friday October 18, 2013 @07:07PM (#6574547)
I did not know that! Thanks, I'll look.
Re:Who exactly is crawling my site? by sergiosgc on Saturday October 19, 2013 @04:04AM (#6575732)
I don't block completely, but have for a long time configured servers to react like this:

- If it's cached content, there's no limit

- If it's non cached content and it is a bot, direct to the pool of servers assigned to uninteresting bots

- If it is an interesting bot (Google, bing, a couple others), use instead a pool of servers for interesting bots

Some bots are not only non-traffic generating, they are absurdly aggressive, consuming 99% of server resources for the hour or so needed to pull the content, regular site users be damned. A configuration like this is reasonably low maintenance while limiting the damage by bots.

Re:Who exactly is crawling my site? by mrgreenfur on Tuesday October 22, 2013 @11:57AM (#6592756)
I'm late, but for this there are companies that sit in the middle and block crawlers, spambots and known-malicious addresses. Here's one I use: http://www.incapsula.com/ So far I've been happy with it!