https://addons.mozilla.org/en-US/firefox/addon/random_user_a...
and then in your browser just type the letter you assign it to: for example, I have 'i' == the searchbox for IMDB, so I type 'i [movie]' in my url and it brings up the IMDB search of that movie . https://i.imgur.com/dXdwsbA.png
So you can just assign 'a' to that search box and type 'a [URL]' in your address bar and it will submit it to your little thing.
If the install is under "eixk3.somedomain.com/ladderforfriends" and it sits behind a reverse proxy, it might as well be invisible to the internet, unless your DNS provider is an idiot and allows zone transfers, or you are on a network where someone is snarfing up DNS requests and then distributing that info to third parties. If you restrict it to TLS 1.3, even someone sniffing traffic from one of your friends won't be able to glean anything useful, because the requested hostname is never sent in plaintext.
Rotate the hostname if/when it becomes necessary...
You could mitigate that with a wildcard cert, but still..
Create a rule to replace user agent with "Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
I just tried it and seems to work.
This piece of crap is, unfortunately, unfit.
Also, how effective is this really? Don’t the big news sites check the IP address of the user agents that claim to be GoogleBot?
They provide ways to verify Googlebot IPs specifically, anyone who cares to check wouldn't be fooled by running a fake Googlebot on Googles cloud.
Likewise with Bingbot: https://www.bing.com/webmasters/help/how-to-verify-bingbot-3...
I also think the antitrust suit (and many more) need to happen for more obvious things like buying out competitors. However, how does publishing a list of valid IPs for their web crawlers constitute anticompetitive behavior? Anyone can publish a similar list, and any company can choose to reference those lists.
https://www.404media.co/google-is-the-only-search-engine-tha...
Whether by IP, user account, user agent, whatever
It may not be the kind of explicit anticompetitive behavior we normally see, but it needs to be regulated on the same grounds.
The website operator can do with that identity as they wish.
They could block it, accept it, accept it but only on Tuesday afternoon.
---
"Anticompetitive" would be some action by Google to suppress competitors. Offering identification is not that.
The law may need more time to catch up, but search indexing will eventually be made a utility.
And that has an argument.
But in the general case no.
It's all the guise of feelgood stuff like "make sure it's google, and not some abusive scraper" language. But the end-result is pretty clear. Just because they have a parallel construction of a valid reason why they're doing something, doesn't mean they don't enjoy the convenient benefits it brings.
It's probably orders of magnitude greater than the revenue loss from the tiny minority of people doing such things, especially given not everyone who uses tools like these will become a subscriber if blocked, so that cuts the "lost" revenue down even further.
edit: Indeed I just tried curling cloudflare.com with Googlebots user agent and they immediately gave me the finger (403) on the very first request.
But perhaps it’s necessary, as it seems Firefox no longer has an about:config option to override the user agent…am I missing it somewhere?