Re:APIs will disrupt RPA. This is obvious to engineers
by rckrd on Friday June 04, 2021 @02:35PM
(#27397274)
Author here. I would say that even though you're grabbing the markup from an API, trying to get structured data from presentation logic still seems closer to RPA than not.
And Wikipedia might be an unique example - the markup structure is fairly stable vs. commercial SaaS.
If a competitor offered API access with the data you wanted, you would always choose the API over screen scraping. If a screen scraping business becomes big enough, they usually either partner with the existing sites (i.e. Plaid) or a competitor does.
I wrote a highly complex analysis engine for Wikipedia and Wikidata that fetched Wiki markup through the API and tried to parse the Wiki markup.
It was big mistake.
The feedback loop that Wikipedia editors work on is "does the HTML output look OK?" and not "does the markup conform to an unknown specification?" so part of the problems is fixing up "errors" that exist against what you think the markup meaning is that don't exist when the document is rendered.
I wrote something that turned the HTML to a DOM tree that could get photo metadata out of a flickr page knowing just a few things about the document and URL structure; I gave up on maintaining the API-based tool and was able to get highly accurate (better than a week of work on the markup parser) extraction of Wikiiedia Commons metadata in about 20 minutes out of the HTML.
As an engineer I also know the business folk tend to design APIs to maximize their business value and minimize my business value. Usually I look at a web site, see information I want to scrape and I "just do it". With an API you tend to make a big investment before you realize exactly how restricted your access is.
Frequently it is "RPA and done" as opposed to "you can't get from here to there" with APIs in commercial practice.