Re:List of apps people pay for but have low rating
by namdnay on Saturday May 01, 2021 @02:47PM
(#27008729)
Websites can choose whether to be scraped or not, and google respects that choice
[ Parent
| on HN ]
Re:List of apps people pay for but have low rating
by k4rli on Saturday May 01, 2021 @02:55PM
(#27008800)
To be fair if being scraped were opt-in (Robots Inclusion Standard?) instead of opt-out, we might have a completely different technological world. Who knows.
Re:List of apps people pay for but have low rating
by intricatedetail on Saturday May 01, 2021 @03:03PM
(#27008876)
By creating an account and using their webmaster tools giving up even more of your data?
Re:List of apps people pay for but have low rating
by mitchdoogle on Saturday May 01, 2021 @03:19PM
(#27009022)
No you just have to put a noindex header on requests to your website
Re:List of apps people pay for but have low rating
by intricatedetail on Saturday May 01, 2021 @04:26PM
(#27009523)
So they still can use up my resources?
Re:List of apps people pay for but have low rating
by Google234 on Saturday May 01, 2021 @04:41PM
(#27009626)
How can they know without looking? Again, you could also use the webmasters tools to block it...
Re:List of apps people pay for but have low rating
by uglygoblin on Saturday May 01, 2021 @05:57PM
(#27010262)
Robots.txt can accomplish the same thing which means they periodically grab a single text file from you.
Re:List of apps people pay for but have low rating
by account42 on Monday May 03, 2021 @10:07AM
(#27025253)
> robots.txt can accomplish the same thing
Not exactly - robots.txt will prevent Google from crawling the page but it might still show up in search results (without a snippet or your page title) based on how others link to it.
Re:List of apps people pay for but have low rating
by Google234 on Sunday May 02, 2021 @06:43AM
(#27013990)
How dare they!
Re:List of apps people pay for but have low rating
by hire_charts on Saturday May 01, 2021 @02:53PM
(#27008778)
It's nice of google to respect the choice to effectively become undiscoverable to anyone searching for your product.
Re:List of apps people pay for but have low rating
by ben_w on Saturday May 01, 2021 @04:09PM
(#27009361)
You sound like you have a third opinion, one which would allow a website to appear on Google in response to user searches without Google knowing anything about what the website contains?
Re:List of apps people pay for but have low rating
by DangitBobby on Saturday May 01, 2021 @07:11PM
(#27010770)
No, they are sarcastically indicating that you don't really have a choice, because if you aren't indexed by Google, practically speaking, you might as well not have a website.