Re:Detecting Chrome headless
by tyingq on Saturday August 05, 2017 @01:24PM
(#14936756)
>You don't usually have nice JSON endpoints, so you'll be doing a lot of HTML parsing
That is changing. Most scrapers haven't caught on, but the more modern things airlines are pushing out (their mobile sites and native mobile apps) often have really nice REST/JSON api interfaces. The scrapers are often still scraping the old desktop site which will be the last to get that underpinning.
Re:Detecting Chrome headless
by dsacco on Saturday August 05, 2017 @02:47PM
(#14937231)
Yes, whenever I'm looking for a source to crawl I prefer mobile applications for precisely this reason. Request signing and certificate pinning are an upfront annoyance, but the maintainability is far higher.
That is changing. Most scrapers haven't caught on, but the more modern things airlines are pushing out (their mobile sites and native mobile apps) often have really nice REST/JSON api interfaces. The scrapers are often still scraping the old desktop site which will be the last to get that underpinning.