News for hackers. Stuff that matters.
Re:Fewer Than Half of Google Searches Now Result in a Click by jefftk on Tuesday August 13, 2019 @09:45PM (#20692031)
I was curious what it was complaining about, since https://henryfjordan.com looks great to me. I tried to run it through Google's "Mobile Friendly Test" but fetching failed [1] because your robots.txt has:

    User-agent: *
    Disallow: /
This would explain why you've gotten zero clicks from Google (or I would guess anyone else's) search results!

On the other hand, it's surprising that you would get a notification if you had crawling disabled. Did you set this robots.txt up recently?

[1] https://search.google.com/test/mobile-friendly?id=97_WUiIxx-...

(Disclosure: I work at Google, commenting only for myself)

Re:Fewer Than Half of Google Searches Now Result in a Click by SahAssar on Wednesday August 14, 2019 @02:19AM (#20693140)
Google seems to see robots.txt as "more what you call guidelines, than actual rules". Sites that block googlebot or all bots with robots.txt still turn up in google searches, just without a description, and are obviously still indexed.
Re:Fewer Than Half of Google Searches Now Result in a Click by jefftk on Wednesday August 14, 2019 @07:19AM (#20694332)
robots.txt is a tool to control crawling, not to specify how you would like your site to be displayed (or not) in search results. If you don't want search engines to include your site, set:

    <meta name="robots" content="noindex">
while to block just Google do:

    <meta name="googlebot" content="noindex">
See https://support.google.com/webmasters/answer/93710

If Googlebot is not respecting robots.txt, and is crawling something it's been instructed not to crawl, let me know and I can file a bug?

(Disclosure: I work for Google but not on Search, speaking only for myself)

Re:Fewer Than Half of Google Searches Now Result in a Click by areyousure on Wednesday August 14, 2019 @10:17AM (#20695786)
But that requires that Googlebot be allowed to crawl the page in robots.txt in the first place.

How do you tell Googlebot to not crawl your site and to not index it either?

Previously, one could use the undocumented "Noindex" directive in robots.txt, but this will be disabled soon: https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...

Re:Fewer Than Half of Google Searches Now Result in a Click by tylerl on Wednesday August 14, 2019 @11:03AM (#20696268)
The bot doesn't need to crawl your site for it to be indexed; it crawls other sites that link to yours.

You can specify your index preferences in Webmaster Tools. Don't know if there's a domain-wide off switch in there, but there probably is.

Re:Fewer Than Half of Google Searches Now Result in a Click by SahAssar on Wednesday August 14, 2019 @06:58PM (#20700889)
Using Webmaster Tools is not a good option since it requires you register with the exact company you are probably trying to not interact with.
Re:Fewer Than Half of Google Searches Now Result in a Click by jefftk on Wednesday August 14, 2019 @10:43AM (#20696039)
The blog post you link has a bunch of alternatives, but I agree they're not great. If there are a lot of webmasters who want to be able to noindex through robots.txt then making the case for adding noindex to the standard would be a good next step.

(Still speaking only for myself)

Re:Fewer Than Half of Google Searches Now Result in a Click by ademarre on Wednesday August 14, 2019 @11:18AM (#20696430)
Googlebot actually used to support a noindex rule in robots.txt, but they are removing it.

https://webmasters.googleblog.com/2019/07/a-note-on-unsuppor...

Re:Fewer Than Half of Google Searches Now Result in a Click by jefftk on Wednesday August 14, 2019 @12:43PM (#20697288)
Yes, that was linked above. It looks like this is part of reducing support to what's in the spec?
Re:Fewer Than Half of Google Searches Now Result in a Click by ademarre on Thursday August 15, 2019 @03:45AM (#20703367)
Oops, yep. I didn't see that context.
Re:Fewer Than Half of Google Searches Now Result in a Click by SahAssar on Wednesday August 14, 2019 @03:56PM (#20699330)
I sent you an email, and I'm posting it here but without identifying info:

---

Hi Jeff,

Thank you for your comment, I'm replying via email to send some info I'd rather not share on HN, but will post the same redacted in HN. I used to (back when starting my web-dev career) run a one man show development team of a web agency and all our development/pre-prod sites (that had to be unauthed) had robots.txt to disallow all bots, but they still popped up in Google. Searching some of the old domains in google I found an example here: http://***.***/***, and attached is an example of it showing up in a SERP and a what the robots.txt looks like (and I'm pretty sure that the robots.txt has looked like that since that page was created).

In this case it is just one page that nobody will care about, and since I'm not working on projects that are open but "robots.txt hidden" anymore I don't know if it is as bad as it used to be, but I regularly see pages with the "No information is available for this page" whose domains have robots.txt's that disallow all bots but still show up in Google.

Please let me know if I missed anything :)

Re:Fewer Than Half of Google Searches Now Result in a Click by jefftk on Wednesday August 14, 2019 @05:24PM (#20700093)
Thanks for sending the screenshot! That site shows up with "no information is available for this page", which means that while robots.txt has disallowed bots from crawling it the page is still linked from other pages that do allow crawling.

The robots.txt protocol gives instructions to crawlers about how they should interact with the site. If you instead want to give instructions to indexes, use the noindex meta tag.

Re:Fewer Than Half of Google Searches Now Result in a Click by SahAssar on Wednesday August 14, 2019 @05:45PM (#20700295)
You're right, I was wrong about how to expect a "Disallow: /" to work. But isn't it sorta odd to have a protocol to control crawling (which is usually done to index) but (almost) require a compliant indexer to crawl all pages to comply with the indexing rules?

In this example the robots.txt has clearly told all bots to not crawl this site, but the only way to read the meta tag (or equivalent header) is to crawl the site. So I assume that in this case google either assumes that it is fine to crawl URL's that it has found elsewhere while ignoring the robots.txt or it assumes that pages disallowed by robots.txt are "open for indexing/linking", which would mean that any page both disallowed by robots.txt and which has a noindex meta tag would still show up, right?

What is the intended behavior if a page is disallowed by robots.txt and still linked by another indexed page? Will it get crawled or just assumed to be okay for indexing/linking? Is there any way to tell Google not to index/link and not to crawl?

Re:Fewer Than Half of Google Searches Now Result in a Click by erik_seaberg on Thursday August 15, 2019 @11:08AM (#20705882)
If you have a calendar where every month links to the previous and next months, a crawler can get stuck and hammer the server. That's the kind of thing robots.txt is for.
Re:Fewer Than Half of Google Searches Now Result in a Click by dorgo on Wednesday August 14, 2019 @02:53AM (#20693263)
>"more what you call guidelines, than actual rules"

they can index without scraping. It is enough that other websites have links to you site. So the google bot follows the rules in robots.txt to the letter. "no-index" is the way to stay away from google.

Re:Fewer Than Half of Google Searches Now Result in a Click by SahAssar on Wednesday August 14, 2019 @07:09PM (#20700945)
They can't read my no-index if they obey my robots.txt. Do they break the robots.txt to be able to read my no-index or do they assume my "Disallow: /" means I'm fine with them indexing/linking?

Without the noindex part of robots.txt (which google decided to ignore not so long ago) this is not solvable.

Re:Fewer Than Half of Google Searches Now Result in a Click by henryfjordan on Wednesday August 14, 2019 @01:18PM (#20697670)
Oh, I just added that yesterday as a response to the email. Before that I was actually running Google Analytics but since I get basically 0 clicks it wasn't really useful.

I have a feeling the PDF viewer triggered it, cause on Mobile it defaults to showing the whole page which results in tiny text but that's easily fixed by the user so I prefer to leave it like that.