Re:Triggering Video Playback with Bots and Crawlers
by maccam94 on Friday December 22, 2017 @07:28PM
(#15992064)
The conclusion that Facebook has bots other than Facebot is interesting. I wonder if Scott was able to identify how many problematic user agent strings were on Facebook's network.
[ Parent
| on HN ]
Re:Triggering Video Playback with Bots and Crawlers
by urlgrey on Friday December 22, 2017 @08:16PM
(#15992302)
Great question! There was a fairly even distribution of slow-performing requests across the many user-agent strings coming from the Facebook ASN.
My hunch is that the Facebook crawlers send user-agent strings that Facebook has seen in requests to their own services. This allows them to crawl content in linked posts masquerading as a device that their users would actually use.
Re:Triggering Video Playback with Bots and Crawlers
by mcphilip on Friday December 22, 2017 @11:03PM
(#15992850)
> My hunch is that the Facebook crawlers send user-agent strings that Facebook has seen in requests to their own services. This allows them to crawl content in linked posts masquerading as a device that their users would actually use.
Very interesting theory, I’d love a follow up post if this can eventually be confirmed or denied :). That would be very clever of FB but very, very rude to all of the small fish they crawl.