Wonder what the story is behind these two...
User-Agent: bender
Disallow: /my_shiny_metal_ass
User-Agent: Gort
Disallow: /earth
Re:Fastly Outage
by gaius_baltar on Tuesday June 08, 2021 @10:14AM
(#27435470)
That's the good old Easter eggs, perhaps a memory from when Reddit was a nice place. They stop appearing and are replaced by dark patterns once sites jump the shark.
I reod some people use false slugs in the robots.txt as a honey pot of sorts. IPs that actually read the robots.txt, ignore the disallow, and still access the uri are outright banned.
Re:Fastly Outage
by taylorfinley on Tuesday June 08, 2021 @01:18PM
(#27438032)
Then when a flamewar breaks out you just have to get your adversary to click a link to get them IP banned.
Ha, that would have been a really smart idea! Sadly we didn't think of that at the time. But we had other honey pot URLs.
It might be related to the time few years ago when Google added exclusions for user agent t1300 in regard to its founders. Gort seems to be a robot from old scifi and bender might be something similar.
Bender is from Futurama, Gort is old classic scifi: https://en.wikipedia.org/wiki/Gort_(The_Day_the_Earth_Stood_...
Just some fun humor we added for other nerds who read robots.txt files.
Easter eggs
It's neckbeard humor.
I'll have you know our beards were neatly trimmed when we added those.
I know you, and I find that hard to believe ;)
> I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results for some queries since nobody is fighting about the position until it's widely known.
That’s not going to happen before Cloudflare is dethroned. See this recent thread for some perspective: https://news.ycombinator.com/item?id=27153603
And even if there’s no Cloudflare, large sites that people want to search will always find ways to block bad bots.
The only thing I can think of that might work is using crowd-sourced data, with all the problems that come with crowdsourcing.
Sadness.
There is a solution for all this mess and I'm blocking HN and a few different domains until I implement at least the first step after which I can share it here.
Re:Fastly Outage
by bruce343434 on Tuesday June 08, 2021 @12:37PM
(#27437481)
try editing your hosts file to redirect reddit to old.reddit
/etc/hosts
reddit.com old.reddit.com
www.reddit.com old.reddit.com
np.reddit.com old.reddit.com
https://old.reddit.com/robots.txt
is very different from this:
https://reddit.com/robots.txt
I guess there is a market for search engine (maybe accessed through tor) which does not care about robots.txt, DMCAs, right to be forgotten etc. Bootstrapping it should not be that hard since it can also provide better results for some queries since nobody is fighting about the position until it's widely known.
I'm not sure how far are we from being able to do full text internet search. Or rather even quote search, preferably some fuzziness options. That would be cool, Google's quotation marks were really neat back when they were working.