News for hackers. Stuff that matters.

A comment on EU Open Web Search project kicked off. Parent comment

Re:EU Open Web Search project kicked off by rgrieselhuber on Tuesday September 20, 2022 @03:16PM (#32916487)
The structured data on sites, unfortunately, still requires a crawler to index that content, which serves as a barrier for search engine startups. At a minimum, adding some metadata content to XML sitemaps would go a long way to solving some of this problem (title, meta description, content summary, even structured data to the sitemaps).
[ Parent | on HN ]
Re:EU Open Web Search project kicked off by DocTomoe on Wednesday September 21, 2022 @02:56AM (#32921621)
Deep down in my soul, the long-locked-away SEO of my money-hustling youth just grinned in anticipation.

We have had embedded metadata in websites for decades. In the beginning, Search Engines did even use them. Until someone started stuffing unrelated keywords in it to rank higher.

[ Parent | on HN ]
Re:EU Open Web Search project kicked off by rrwo on Wednesday September 21, 2022 @07:35AM (#32923584)
> The structured data on sites, unfortunately, still requires a crawler to index that content

How is that any different from requiring a crawler to index XML sitemaps?

> At a minimum, adding some metadata content to XML sitemaps

The purpose of a sitemap is to tell a web robot what resources there are, with some minimal metadata about page titles and last modified date.

Google has some extensions for identifying images and videos.

But that adds more work for site maintainers, who have to duplicate work.

[ Parent | on HN ]
Re:EU Open Web Search project kicked off by Eduard on Tuesday September 20, 2022 @05:02PM (#32917759)
What's the problem of using any of the many free webcrawler (libraries) available to crawl a website (even if solely based on the pages advertised by sitemap.xml / robots.txt-announced sitemaps), then extract structured data from these pages?

I don't see this as a barrier unique to startups.

[ Parent | on HN ]
Re:EU Open Web Search project kicked off by rgrieselhuber on Tuesday September 20, 2022 @05:05PM (#32917784)
It's easy to do for small sets of sites, but try doing this at web-scale and you quickly run into a large financial barrier. It's not about technical feasibility as much as it is cost.
[ Parent | on HN ]