Some web firewalls now even detect fake crawlers by using instructions published by search engines (Google: https://developers.google.com/search/docs/advanced/crawling/..., Microsoft: https://www.bing.com/webmasters/help/how-to-verify-bingbot-3..., Yandex: https://yandex.com/support/webmaster/robot-workings/check-ya..., Apple: restrict to 17.0.0.0/8 (their IP range, https://support.apple.com/en-us/HT204683))