Re:Tips for reliable web automation and scraping selectors
by imgabe on Monday February 01, 2021 @07:30PM
(#25996045)
Another tip I've found extremely helpful for webscraping: check the <head> for <meta> tags or a <script type="application/ld+json"> tag that might already have the information you want collected neatly in one place. You may be able to save yourself a lot of time and grief.
[ Parent
| on HN ]
Re:Tips for reliable web automation and scraping selectors
by manojlds on Tuesday February 02, 2021 @01:43AM
(#25998009)
We built a library for extracting these data - https://github.com/indix/web-auto-extractor
Re:Tips for reliable web automation and scraping selectors
by JimDabell on Tuesday February 02, 2021 @01:48AM
(#25998032)
Also, if the site is based on WordPress, the API is often open for read-only access, so you can fetch richer information and you won’t have to parse the full HTML document to get the content in question.
Re:Tips for reliable web automation and scraping selectors
by liquorice on Tuesday February 02, 2021 @05:46AM
(#25999427)
Unfortunately this data is often inconsistent with the visual representation. For example, webshops often list their product as 'InStock' regardless of the actual stock status. Since products are in stock most of the time, you will not find out about this and thus likely extract wrong data in the future.
This was especially apparent when I tried to get my hands on some weights during the early corona pandemic. All webshops were out of stock, but in about 80% of them the schema markup indicated otherwise
Re:Tips for reliable web automation and scraping selectors
by tschiller on Monday February 01, 2021 @08:19PM
(#25996424)
That's definitely the easiest when it's there. In some cases the microdata will instead be embedded in the HTML tags in the body: https://schema.org/docs/gs.html