News for hackers. Stuff that matters.
Re:Twitter starts to require login to view tweets by johnc1231 on Tuesday August 24, 2021 @11:03AM (#28289763)
How do they identify a crawler? Can a user not masquerade as a crawler somehow?
Re:Twitter starts to require login to view tweets by est31 on Tuesday August 24, 2021 @11:14AM (#28289928)
Sometimes they use user agents for this, but those are easily faked, so it's only done by websites that don't have comprehensive auth walls.

A more comprehensive method is based on ip ranges, say whitelisting traffic from Google and Bing. This gives you > 95% of search traffic as Google alone has >90%, and many various smaller search engines like Yahoo or DDG are Bing resellers.

On the other hand, a pure ip based check can be circumvented too. Sometimes you can view how search engines see a website through google translate. But places like LinkedIn have countermeasures for all of these circumventions.

Re:Twitter starts to require login to view tweets by tjpnz on Tuesday August 24, 2021 @11:07AM (#28289826)
Normally by user agent. You can adopt the same UA to get past some paywalls, but it doesn't always work.