Thanks. This is interesting! I would like to try it out. I'm doing scraping now, because our vendors refuse to give us API or they don't have one. They only give us data feeds via email. And the feeds are very sketchy, no images for example. We have to scrape their websites. My main tool is puppeteer, However, I found it's way easier to scrape XHR network responses, global javascript objects or look into <script type="application/json">, not from the UI.

In my case, I need to scrape 2 things. One is the sku number, which is often not displayed on the ui, but if you look into the source code of the webpage, you can usually find it.

I also need to scrape all product variants, such as all colors and product sizes, all finishes, etc.

If I do this via UI, I need to click each product option, and wait for the ui change and then scrape the content. My work becomes easier if I can capture a json data exchange or a javascript object that contains this information. For most of the cases, I can find it. So I gave up on reading things directly from the UI.

I learned a new word, "ETL workflow". I have been doing this data scraping thing for a while, I didn't know there is a whole ecosystem existing. Scraping is such a pain, it's sad that there is a lack of common data exchange standards.