Twitter cut off the ability to read a tweet by fetching its URL with a HTTP GET

Re:Twitter cut off the ability to read a tweet by fetching i... by smarx007 on Friday December 18, 2020 @06:35AM (#25466105)

I think Google shares IP range blocks so you could implement a check like "if(isGooglebot(user_agent) && isGooglebotIp(ip_addr))" in your system.

Edit: ah no, they stopped https://developers.google.com/search/docs/advanced/crawling/.... I don't think 2 DNS lookups are acceptable to block a GET request but I think it can be done out of band, i.e. isGooglebotIp function can fire away a Redis query and if nothing was found, to put the ip into the DNS verify queue. A few requests later, the bot will now get banned thanks to a new record in Redis.

[ Parent | on HN ]

Re:Twitter cut off the ability to read a tweet by fetching i... by 1vuio0pswjnm7 on Friday December 18, 2020 @04:31PM (#25472318)

No need to use a Googlebot UA string. Others will work. Such as

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0

[ Parent | on HN ]