X updates its Terms to prohibit crawling/scraping of its data

Re:X updates its Terms to prohibit crawling/scraping of its ... by grumple on Friday September 08, 2023 @10:39AM (#37434096)

Won't this robots.txt update stop major search engines from indexing Twitter? Seems like a great way to kill revenue as Twitter vanishes from search results.

[ Parent | on HN ]

Re:X updates its Terms to prohibit crawling/scraping of its ... by jraph on Friday September 08, 2023 @10:46AM (#37434173)

I wonder if search engines actually use rules for Googlebot when parsing some (specific?) robots.txt.

"Googlebot" might as well be the "Mozilla/5.0" of search index bot user agents.

[ Parent | on HN ]

Re:X updates its Terms to prohibit crawling/scraping of its ... by wiktor-k on Friday September 08, 2023 @10:47AM (#37434182)

Twitter is benefiting from network effects by now. I'm not even sure when have I encountered a tweet in search results last time.

[ Parent | on HN ]

Re:X updates its Terms to prohibit crawling/scraping of its ... by jeroenhd on Friday September 08, 2023 @10:57AM (#37434338)

I encounter tweets all the time when I'm trying to find out more about recent events.

[ Parent | on HN ]

Re:X updates its Terms to prohibit crawling/scraping of its ... by thisismyhna on Friday September 08, 2023 @10:47AM (#37434191)

Does twitter rely on that? Most of the time I go to twitter is from news stories, comments or other social media.

[ Parent | on HN ]

Re:X updates its Terms to prohibit crawling/scraping of its ... by sundaeofshock on Friday September 08, 2023 @10:51AM (#37434247)

Any revenue damage from this will be faulty minor compared to the damage that has already been done.

[ Parent | on HN ]

Re:X updates its Terms to prohibit crawling/scraping of its ... by dahfizz on Friday September 08, 2023 @03:07PM (#37437982)

They have a section in the robots.txt that allows Google. They must think no other search engine matters.

[ Parent | on HN ]