> Why is Google the only one who can do it? Must they have yet another monopoly?

I run a search engine and do my own crawling, and this does not correspond to my view of reality at all.

I have had almost no problems with getting blocked. If I have gotten blocked, it's usually been my own fault and I've been able to get unblocked by sending them an email explaining I run a search engine and asking for forgiveness because my bot wasn't behaving well.

The bots that do get blocked are bots in most cases bots that misbehave, ignore robots.txt, fetch the same resources repeatedly or with insane crawl-delays.

There are a few rare exceptions, but the whole "why does Google get a free pass when I don't?"-angle just doesn't hold water at all.

> I've had scripts that made 1 request per day get blocked for no reason. Not to mention the endless cloudflare javascript bullshit they made me support for it to even work.

Google doesn't repeat requests every day. I don't repeat requests every day. That's a weird thing to do, and it's well within a site owner's prerogative to block that nonsense.

> Google doesn't repeat requests every day. I don't repeat requests every day.

I scrape one site whose content changes every second. How is it "nonsense" to make one request every 24 hours? I make hundreds, thousands when I browse their site normally using my browser.

We've got people in this very thread talking about bots making hundreds of requests per second. How is one request every 24 hours harming anyone? People told me to make one request per hour to avoid hammering their servers, I decided to wait a day instead. It boggles my mind that this generous interval could possibly be considered abuse. How long should the interval be then? A month? A year? Infinitely long so the scraper never makes requests?

I built a search engine in school. I was lazy and stupid with the scraper and ended up writing a bug that caused it to loop on certain sites. That lead to the entire Duke law school site being DOS's during their class sign up period. Sorry for ruining scraping for everyone, but this is why websites don't want people scraping them.

But you have no freedom when contacting my servers. I can send you 403's just because I don't like your face. There is zero entitlement that you have any access to my servers in any way I don't permit. If I say no automated access then on what grounds to you have to do it anyway?

> Why is Google the only one who can do it?

Because site operators explicitly allow them automated access. If you want the same treatment you have to ask for it.

> I can send you 403's just because I don't like your face.

Sure. At least then you're being honest. If you hate me, it doesn't matter what user agent I use to access your site. Browsers, scripts, they are all me.

Why do you think it’s dishonest to send 403s to bots but not browsers? Method of access matters — you the human might have access to your safety deposit box but the bank is still allowed to make rules about your access — like you have to come during business hours and you can’t send someone on your behalf.

> you can’t send someone on your behalf

I totally can though. If sign a document saying another person can do such and such on my behalf, that person can totally do that. Yes, even at the bank. No different from a user agent, really.

> Why is Google the only one who can do it? Must they have yet another monopoly?

Google can do it because most website operators want them to index their site. Plus, it is trivial to tell google to stop. That goes for all search engines.