News for hackers. Stuff that matters.

A comment on EU Open Web Search project kicked off. Parent comment

Re:EU Open Web Search project kicked off by Eduard on Tuesday September 20, 2022 @05:02PM (#32917759)
What's the problem of using any of the many free webcrawler (libraries) available to crawl a website (even if solely based on the pages advertised by sitemap.xml / robots.txt-announced sitemaps), then extract structured data from these pages?

I don't see this as a barrier unique to startups.

[ Parent | on HN ]
Re:EU Open Web Search project kicked off by rgrieselhuber on Tuesday September 20, 2022 @05:05PM (#32917784)
It's easy to do for small sets of sites, but try doing this at web-scale and you quickly run into a large financial barrier. It's not about technical feasibility as much as it is cost.
[ Parent | on HN ]