Re:Google will punish sites that use annoying pop-up ads
by squeaky-clean on Sunday September 10, 2017 @01:58PM
(#15213368)
It's not illegal, but if you're hiding content from Googlebot because you don't want it to derank that page, once they crawl it with an agent that hides the fact that it's Google, they will then derank your whole website.
So it's not illegal, but if you think you're beating Google by doing it, they're well aware of this workaround and will punish you harder.
If you don't care about ranking, then no biggie. But why would you manipulate your web server like that if you weren't trying to rank.
Re:Google will punish sites that use annoying pop-up ads
by falcor84 on Sunday September 10, 2017 @02:31PM
(#15213521)
>scrape my site without permission.
No one needs permission to request a web page, and that includes bots. But if that still bothers you, why not create arobots.txt with your preferences?
Re:Google will punish sites that use annoying pop-up ads
by Karunamon on Sunday September 10, 2017 @02:06PM
(#15213409)
You’re giving permission to scrape by way of having a page accessible to the general public. If googlebot is bothering you so much, install a two line robots.txt and it’ll go away.
I know that there are practical concerns here, but Google has done a masterful job of shaping the conversation so that many people think anything not sanctioned by Almighty Google with regard to SEO (let alone actual "blackhat" tactics) is illegal or immoral.