News for hackers. Stuff that matters.
Re:Google will punish sites that use annoying pop-up ads by ryanwaggoner on Sunday September 10, 2017 @01:29PM (#15213224)
You're not getting "caught", and Google can fuck right off with any "terms of service" that I never agreed to about how I respond when they scrape my site without permission.

I know that there are practical concerns here, but Google has done a masterful job of shaping the conversation so that many people think anything not sanctioned by Almighty Google with regard to SEO (let alone actual "blackhat" tactics) is illegal or immoral.

Re:Google will punish sites that use annoying pop-up ads by squeaky-clean on Sunday September 10, 2017 @01:58PM (#15213368)
It's not illegal, but if you're hiding content from Googlebot because you don't want it to derank that page, once they crawl it with an agent that hides the fact that it's Google, they will then derank your whole website.

So it's not illegal, but if you think you're beating Google by doing it, they're well aware of this workaround and will punish you harder.

If you don't care about ranking, then no biggie. But why would you manipulate your web server like that if you weren't trying to rank.

Re:Google will punish sites that use annoying pop-up ads by falcor84 on Sunday September 10, 2017 @02:31PM (#15213521)
>scrape my site without permission.

No one needs permission to request a web page, and that includes bots. But if that still bothers you, why not create arobots.txt with your preferences?

Re:Google will punish sites that use annoying pop-up ads by Karunamon on Sunday September 10, 2017 @02:06PM (#15213409)
You’re giving permission to scrape by way of having a page accessible to the general public. If googlebot is bothering you so much, install a two line robots.txt and it’ll go away.
Re:Google will punish sites that use annoying pop-up ads by jiggunjer on Sunday September 10, 2017 @10:34PM (#15215503)
robots can ignore your /robots.txt.
Re:Google will punish sites that use annoying pop-up ads by Karunamon on Monday September 11, 2017 @10:38AM (#15218816)
Googlebot, and every other search engine robot of note. Putting a page up on the public internet includes the risk that it may be visited.