I'm not surprised. Some people think humans read robots.txt and get super angry when the crawler doesn't understand.

I read robots.txt, but I'm not a massive corporation.

> (absl::StartsWithIgnoreCase(key, "disallaw")))));

Ah, the southern version. :)

This is great. #kAllowFrequentTypos

Ive never made a tpyo in my life!!!

Whats a tpyo? My typo array for "typo" only has tipo, typpo and thai-pho

If you're gonna include thai-pho you also need to include thai-fur...

Makes sense because not everyone speaks English as native language. Disalow is pretty close to disallow, phonetically.

Yuck though! Imagine if you were writing a compiler. Would you make it accept “unsinged” “unnsigned” “unssined” and “unsined” as keywords, just to catch spelling mistakes? Not sure I like that pattern.

It's a little different in that case, since the person using the parser is also the person writing the input to the parser. So if the input fails the parser, the author of the code can simply correct it. As I understand it, there's no single standard that captures how all robots.txt files are formatted, so there's no "standard parser" that the authors of these files could be expected to pass.

That is not an excuse. Non-native speakers can learn to spell.