Re:Experimental library for scraping websites using OpenAI's...
by nghota on Saturday March 25, 2023 @03:27PM
(#35306209)
Do you really need GPT for this? - see https://nghota.com for an api that provides something similar but for articles (I am the developer there!).
[ Parent
| on HN ]
Re:Experimental library for scraping websites using OpenAI's...
by PUSH_AX on Saturday March 25, 2023 @03:41PM
(#35306378)
> Do you really need GPT for this?
Objectively, if you want something meaningful back, yes, you do.
Re:Experimental library for scraping websites using OpenAI's...
by chhenning on Saturday March 25, 2023 @03:34PM
(#35306284)
All I got is this:
```json { "url": "https://www.3sonsbrewingco.com/menus", "title": "MENU | 3sons", "content": " \r\n\r\nBrewery & Kitchen\r\n\r\nEAT & DRINK\r\n\r\n " } ```
I was hoping for some menu items...
Re:Experimental library for scraping websites using OpenAI's...
by dopidopHN on Saturday March 25, 2023 @03:51PM
(#35306514)
Can you refine further ? Because indeed that look like something beautiful soup would output
Re:Experimental library for scraping websites using OpenAI's...
by nghota on Saturday March 25, 2023 @04:05PM
(#35306710)
Only articles are supported atm. I am working on algorithms for other page types.
Re:Experimental library for scraping websites using OpenAI's...
by ushakov on Saturday March 25, 2023 @03:46PM
(#35306440)
There’s also Apify(.com)
Re:Experimental library for scraping websites using OpenAI's...
by tomberin on Saturday March 25, 2023 @03:50PM
(#35306495)
Perhaps not, the author mentioned on Mastodon that he was exploring simpler models.
Re:Experimental library for scraping websites using OpenAI's...
by joegibbs on Saturday March 25, 2023 @08:14PM
(#35309050)
I think so. We use GPT for stuff like extracting the author from articles (if they aren't in the Schema.org data or marked up elsewhere), summarising them, extracting relevant tags to link articles together, etc. It's very useful for that kind of information extraction stuff when there's no structure to the data, or the structure is only sometimes followed.