News for hackers. Stuff that matters.

A comment on Web scraping with GPT-4o: powerful but expensive. Parent comment

Re:Web scraping with GPT-4o: powerful but expensive by sebstefan on Tuesday September 03, 2024 @07:54AM (#41434003)
The LLM is resistant to website updates that would break normal scraping

If you do like the author did and ask it to generate xPaths, you can use it once, use the xPaths it generated for regular scraping, then once it breaks fall back to the LLM to update the xPaths and fall back one more time to alerting a human if the data doesn't start flowing again, or if something breaks further down the pipeline because the data is in an unexpected format.

[ Parent | on HN ]
Re:Web scraping with GPT-4o: powerful but expensive by is_true on Tuesday September 03, 2024 @08:49AM (#41434458)
Xpath can be based on content, not only positions
[ Parent | on HN ]
Re:Web scraping with GPT-4o: powerful but expensive by sebstefan on Wednesday September 04, 2024 @02:40AM (#41442555)
I normally use query selectors for scraping, I'm not sure why the author doesn't
[ Parent | on HN ]
Re:Web scraping with GPT-4o: powerful but expensive by melenaboija on Tuesday September 03, 2024 @08:49AM (#41434460)
Unless the update is splitting cells.

> Turns out, a simple table from Wikipedia (Human development index) breaks the model because rows with repeated values are merged

[ Parent | on HN ]
Re:Web scraping with GPT-4o: powerful but expensive by sebstefan on Tuesday September 03, 2024 @09:58AM (#41435020)
Nah, still correct :-) that would break the regular scraping as well

>The LLM is resistant to website updates that would break normal scraping

[ Parent | on HN ]
Re:Web scraping with GPT-4o: powerful but expensive by trog on Tuesday September 03, 2024 @06:28PM (#41439968)
This is absolutely true, but it does have to be weighed against the performance benefits of something that doesn't require invoking an LLM to operate.

If the cost of updating some xPath things every now and then is relatively low - which I guess means "your target site is not actively & deliberately obfuscating their website specifically to stop people scraping it"), running a basic xPath scraper would be maybe multiple orders of magnitude more efficient.

Using LLMs to monitor the changes and generate new xPaths is an awesome idea though - it takes the expensive part of the process and (hopefully) automates it away, so you get the benefits of both worlds.

[ Parent | on HN ]