News for hackers. Stuff that matters.
Re:Web scraping with GPT-4o: powerful but expensive by ErikAugust on Monday September 02, 2024 @07:39PM (#41429712)
Running it through Readability:

https://github.com/mozilla/readability

Re:Web scraping with GPT-4o: powerful but expensive by parhamn on Monday September 02, 2024 @07:44PM (#41429738)
I snuck in an edit about readability before I saw your reply. The quality of that one in particular is very meh, especially for most new sites and then you lose all of the dom structure in case you want to do more with the page. Though now I'm curious how it works on the weather.com page the author tried. pupeteer -> screenshot -> ocr (or even multi-modal which many do OCR first) -> LLM pipeline might work better there.
Re:Web scraping with GPT-4o: powerful but expensive by LunaSea on Tuesday September 03, 2024 @03:21AM (#41432210)
Issue is that Llama are x100 more expensive at the very least.