News for hackers. Stuff that matters.
Re:The Lose-Lose-Lose-Lose Bill C-18 Outcome: Meta Blocking ... by zlg_codes on Monday August 07, 2023 @03:32PM (#37039720)
robots.txt is not an actual Internet standard, and there are no standard controls for when a bot ignores its contents. You're on your own to protect your pages.

Granted, I have `disallow: /` in mine because I don't want my stuff scraped, but I still see Googlebot sometimes, and others, try to crawl my site.

It's not a very effective opt-out, because it requires the 'attacker' to honor the file's settings.

Feel free to enable it on your own server and watch the logs for a few days after sharing some links.

Re:The Lose-Lose-Lose-Lose Bill C-18 Outcome: Meta Blocking ... by 18pfsmt on Tuesday August 08, 2023 @07:54PM (#37057046)
Have you tried crying about the fact that you are getting traffic to your website that nobody would see otherwise?

I hope daddy government treats you well.

Count on Big Brother to save you, and they will beat you down when it's time as well.