Twitter cut off the ability to read a tweet by fetching its URL with a HTTP GET

Re:Twitter cut off the ability to read a tweet by fetching i... by antpls on Friday December 18, 2020 @06:15AM (#25465990)

Given that trick is spreading for several sites now, the trick won't last long. Google could for example generate secret unique user agents for the biggest players. Biggest players would then only allow requests from that secret unique UA.

[ Parent | on HN ]

Re:Twitter cut off the ability to read a tweet by fetching i... by smarx007 on Friday December 18, 2020 @06:35AM (#25466105)

I think Google shares IP range blocks so you could implement a check like "if(isGooglebot(user_agent) && isGooglebotIp(ip_addr))" in your system.

Edit: ah no, they stopped https://developers.google.com/search/docs/advanced/crawling/.... I don't think 2 DNS lookups are acceptable to block a GET request but I think it can be done out of band, i.e. isGooglebotIp function can fire away a Redis query and if nothing was found, to put the ip into the DNS verify queue. A few requests later, the bot will now get banned thanks to a new record in Redis.

[ Parent | on HN ]

Re:Twitter cut off the ability to read a tweet by fetching i... by 1vuio0pswjnm7 on Friday December 18, 2020 @04:31PM (#25472318)

No need to use a Googlebot UA string. Others will work. Such as

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Firefox/78.0

[ Parent | on HN ]

Re:Twitter cut off the ability to read a tweet by fetching i... by miyuru on Friday December 18, 2020 @06:40AM (#25466128)

no need a secret unique UA. just PTR the IP and check the host.

This method is already used by stackoverflow to hide the sitemap.

https://meta.stackexchange.com/a/37272/158100

[ Parent | on HN ]

Re:Twitter cut off the ability to read a tweet by fetching i... by smileysteve on Friday December 18, 2020 @08:42AM (#25466937)

Things that would be inadvisable while most of these companies are actively under suit for antitrust issues.

[ Parent | on HN ]

Re:Twitter cut off the ability to read a tweet by fetching i... by kobalsky on Friday December 18, 2020 @04:46PM (#25472453)

it already happens, try to create your own search engine an index Amazon. captchas everywhere, their robots.txt is just for show.

[ Parent | on HN ]

Re:Twitter cut off the ability to read a tweet by fetching i... by Too on Monday December 21, 2020 @05:24AM (#25493620)

This trick is almost older than the internet, so if someone cared they would block it already. Sending google.com as Referrer is another variant of it. Before stackoverflow days this was very useful for getting past the paywall on expertsexchange for example.

I was under the impression that serving other content to google would greatly punish your pagerank and even pull you off the search results completely.

[ Parent | on HN ]

Re:Twitter cut off the ability to read a tweet by fetching i... by txdv on Friday December 18, 2020 @06:23AM (#25466039)

so much to google embracing the open web

[ Parent | on HN ]