News for hackers. Stuff that matters.
Re:The joys and sorrows of maintaining a personal website by AshamedCaptain on Friday May 06, 2022 @04:40PM (#31289418)
> The enemy possesses every advantage. Personal website owners are nearly always just ordinary individuals. They often have glacially slow upload bandwidths, [...]. The enemy has seemingly unlimited funds, massive resources, acres of server farms,

One example of this is Googlebot "attacks". I have a personal, mostly static website that I've been serving from a home connection for over 25 years, upgrading the hardware and moving it with me as I relocated. Out of these 25 years, Google has been number #1 user of bandwidth, CPU time, and as a consequence the #1 generator of headaches.

During the mid 2000s I had numerous problems keeping the site online since the Google crawler would just effectively DDoS it, sending tens of concurrents requests completely ignoring the Crawl-Delay in robots.txt (and most of robots.txt, fwiw). Mind you, this was not the most powerful hardware you could find, since it had to be quite low power to be left 24h on, but it was perfectly able to serve the handful of connections per hour I expect out of my website.

Obviously your only alternative is to just ban the entire Google IP range and as a consequence disappear from the Internet. Nowadays the problem is less noticeable because my 1-2W server can handle the onslaught brought by Googlebot just fine, but they still ignore the Crawl-Delay.

Re:The joys and sorrows of maintaining a personal website by freedomben on Friday May 06, 2022 @10:20PM (#31291632)
From what I've been told, the Google Bot absolutely tries to respect the robots.txt. The advice I got was to double check that my syntax and expressions were really correct. A lot of people screw it up so it has to be basically perfect otherwise it won't be parsed correctly. Are you willing to share your robots.txt file?
Re:The joys and sorrows of maintaining a personal website by franze on Saturday May 07, 2022 @04:04AM (#31292922)
Googlebot supports their set of robots.txt instructions https://developers.google.com/search/docs/advanced/robots/cr...

Crawl Delay is not part of it.

Re:The joys and sorrows of maintaining a personal website by idoubtit on Saturday May 07, 2022 @04:18AM (#31292971)
The technical solution is to apply a rate limit on your web server. Then Google bots will receive a HTTP status code 429 "too many requests" that will force them to slow down.

Nginx has `limit_req_zone` that allows fine-tuning the rate limits.

Re:The joys and sorrows of maintaining a personal website by jamesmishra on Friday May 06, 2022 @07:06PM (#31290464)
Why not use something like Cloudflare as a caching layer between you and Google?
Re:The joys and sorrows of maintaining a personal website by rikroots on Saturday May 07, 2022 @07:17AM (#31293640)
Quote from the article:

"From time to time, I consider shielding my website with Cloudflare's services. I resist the temptation, because I strongly suspect that, despite Cloudflare's current good intentions, in the long-run, this is playing into the hands of the corporate-controlled web. Relying on commercial web services limits the freedom we have as personal website owners. Once they have us relying on them, we are forced to play by their rules, and at that point the battle is over. We may not have realized it yet, but we have been captured. Unless we manage to free ourselves, in a short time, we will be working for the enemy. [...]"

Re:The joys and sorrows of maintaining a personal website by iggldiggl on Saturday May 07, 2022 @11:44AM (#31295475)
Just right within the last few days Cloudflare's DDoS protection has started blocking older and/or niche browsers, so all I can say about them is #*'@~?!§%$§#@!!
Re:The joys and sorrows of maintaining a personal website by jgrahamc on Saturday May 07, 2022 @11:52AM (#31295554)
What are we blocking? Can you email details to me? ([email protected])
Re:The joys and sorrows of maintaining a personal website by iggldiggl on Saturday May 07, 2022 @02:32PM (#31297201)
Thank you, reply sent.
Re:The joys and sorrows of maintaining a personal website by Snowworm on Saturday May 07, 2022 @03:11PM (#31297577)
Unrelated, but that is a hilarious username.
Re:The joys and sorrows of maintaining a personal website by bathtub365 on Friday May 06, 2022 @09:44PM (#31291445)
Because Google is the one that needs to change.
Re:The joys and sorrows of maintaining a personal website by freedomben on Friday May 06, 2022 @10:24PM (#31291650)
While that may very well be true, it's also wildly impractical. I would imagine the outcome of that stance will be the skeleton meme "still waiting and google still DDoSes my site."
Re:The joys and sorrows of maintaining a personal website by coryrc on Friday May 06, 2022 @11:02PM (#31291804)
That company did not exist until several years after the events in the story occurred.
Re:The joys and sorrows of maintaining a personal website by franze on Saturday May 07, 2022 @04:01AM (#31292909)
Google does not support crawl delay.

You can set your crawl rate lower every 90 days here https://www.google.com/webmasters/tools/settings

other possibilities are here https://support.google.com/webmasters/answer/48620?hl=en there is a way to lower crawling without total blocking (returning HTTP 503 if there is too much load)

Re:The joys and sorrows of maintaining a personal website by sorokod on Saturday May 07, 2022 @04:12AM (#31292954)
I know it amounts to same, but from the users perspective it would be more accurate to say that Google doesn't respect crawl delay.
Re:The joys and sorrows of maintaining a personal website by franze on Saturday May 07, 2022 @05:38AM (#31293233)
crawl delay was an extension of /robots.txt and never a part of the original "recommendation"

and also never made it into the standard draft https://datatracker.ietf.org/doc/html/draft-rep-wg-topic-00#...

Re:The joys and sorrows of maintaining a personal website by hinkley on Friday May 06, 2022 @05:47PM (#31289911)
I really wonder if this is an area that would be best served by creating a cooperative, allowing you to spread some of this knowledge and troubleshooting across a larger number of humans.

But that's fundamentally a social problem, and the field of software is nothing if not consistent in its resolve to try every other solution before resorting to emotional work. Co-ops that try to come up with a set of rules to handle all situations end up failing, because rules-lawyering turns into a game to be won by some people, at the expense of everyone else (see also, HOAs).

Re:The joys and sorrows of maintaining a personal website by jrochkind1 on Friday May 06, 2022 @07:06PM (#31290472)
Odd, I have always seen the google bot respect the robots.txt. If I disallowed it, it disappeared from my traffic.

Of course, generally people do not want to do this, as they want to appear in google.