There are already sitemaps, and pages used structured data like HTML5/ARIA roles, RDF or JSON+LD to provide some semantic annotations.

I'd rather that web robots use this information to build useful indexes than to have to worry about generating yet another feed in the hopes that it helps people find my content in a search engine.

Besides, a web robot can determine how much other sites link to my content and help determine its overall ranking in results. Adding another type of index file to my site will do nothing to determine how it relates to other sites.

The structured data on sites, unfortunately, still requires a crawler to index that content, which serves as a barrier for search engine startups. At a minimum, adding some metadata content to XML sitemaps would go a long way to solving some of this problem (title, meta description, content summary, even structured data to the sitemaps).

Deep down in my soul, the long-locked-away SEO of my money-hustling youth just grinned in anticipation.

We have had embedded metadata in websites for decades. In the beginning, Search Engines did even use them. Until someone started stuffing unrelated keywords in it to rank higher.

> The structured data on sites, unfortunately, still requires a crawler to index that content

How is that any different from requiring a crawler to index XML sitemaps?

> At a minimum, adding some metadata content to XML sitemaps

The purpose of a sitemap is to tell a web robot what resources there are, with some minimal metadata about page titles and last modified date.

Google has some extensions for identifying images and videos.

But that adds more work for site maintainers, who have to duplicate work.

What's the problem of using any of the many free webcrawler (libraries) available to crawl a website (even if solely based on the pages advertised by sitemap.xml / robots.txt-announced sitemaps), then extract structured data from these pages?

I don't see this as a barrier unique to startups.

It's easy to do for small sets of sites, but try doing this at web-scale and you quickly run into a large financial barrier. It's not about technical feasibility as much as it is cost.

Standard for this already exists [1] but it does not solve the problems of

1. Implementation (sites do not need to have a sitemap; or those that have it, may not have an accurate one)

2. Discoverability (finding sites in the first place, you'll need a centralised directory of all sites; or resort back to crawling in which case sitemaps are not needed)

3. Ranking (biggest problem in creating a search engine)

[1] https://www.sitemaps.org/protocol.html

The sitemaps standard (if this is the basis) would need to be expanded to support additional metadata / structured data to support this idea.

1. This would be up to sites, to your point, major question would be best way to create incentives.

2. This is solvable via a number of approaches, but the search engines themselves would be mostly responsible for finding the right approach for their business. I know how I would do it.

3. Indeed, which would be the main point of this decentralization, to let search engines focus on their hardest problem.

Edit: would Kagi not benefit from having to worry about crawling / indexing sites?

> would Kagi not benefit from having to worry about crawling / indexing sites?

It would, but sitemaps do not provide that function as we discussed above. However if EU Open Web Search succeeded, that is something we could probably use to some extent.

Or use to extend

One problem with that is now you have to trust the websites to give an accurate index of their content.

Anyone who thinks this will work has never tried to index a site. A huge amount of effort is spent trying to figure out if the site is serving different content to users vs crawlers, or if the site is coded to appear visually different to humans vs machines. If you ask sites to index themselves you will get lies only.

I index sites all the time and I think it could work. There will be other problems, of course, but we already are partly there with XML sitemaps. Relying on the large search engines to enforce “honesty” from websites puts them into a mediator role that has a number of negative effects both for search in general and, increasingly, society at large.

Relying on sites to be honest about themselves, is even less likely. There are monetary incentives for many of them not to do that. Many sites host dishonest and clickbait content with extreme levels of SEO already. The cost of dishonesty decreases if you can directly modify the index.

I think that is primarily a symptom of the fact that we have a bottleneck on search interface providers. If it were easier / cheaper for new search engines / rankers to exist in the market, they could fairly easily filter out unscrupulous domains.

I've run a web-scale search engine and I don't think it will work.

Not only are some sites malicious -- mostly unimportant ones -- but many good sites are simply incompetent.

Indeed

I suspect you underestimate how much of the power of search engines is being able to interpret search queries and figure out what a user is really looking for. Even if there were a public, standardised up-to-date high performance full-text index of the entire web freely available I'm willing to bet Google search would be a useful value-add in its ability to answer natural language queries.

> you underestimate how much of the power of search engines is being able to interpret search queries and figure out what a user is really looking for.

So you mean that a search engine is supposed to ignore what you are asking for and instead give you what it thinks you really meant?

If you search for [what year was queen Elizabeth born], Google doesn't return pages that have that phrase in it, rather it returns pages that have an answer to that question. You could call that "ignoring" if you like, but it's what 99% of users expect.

I run an SEO platform SaaS, so I'm familiar. :)

I resort to "natural language queries" only in desperation, when queries that are lists of search-terms have failed.

Actually, they aren't really natural language queries. They are just ordered lists of search-terms. Goo provides no mechanism for saying "This is an english-language question". And even if Goo could parse my natural language, and rephrase it as something like "Are you looking for a list of books published by Douglas Hofstadter?", when you turn that into a query on the index, it stops having anything to do with natural language.

A "dumb" search that just took a search phrase like "books published by Douglas Hofstadter" is going to return pages that have that phrase in it, or something close to that phrase. Google will prioritise results that actually contain such lists, regardless of whether the page contains a phrase like that (e.g. the word "published" is basically ignored by Google). That's all I meant.

We will explore that idea in the project, I also think it may help (but vulnerable for Web index spam by adversary parties).

That is indeed the biggest problem but maybe something that can be more effectively dealt with downstream by the content rankers and potentially even the user base / custom search algorithm builders. Brave's Goggles project is a good early prototype of this concept.

I'm pretty sure we tried that way back in the day with <meta name="keywords" content="spam spam spam spam">. People would stuff that with every word in the english language. Older search engines that used those keywords returned some pretty awful results. You simply can't trust sites, who have a strong incentive to get to the top of SEO rankings, to not lie. In fact, given at least one of your competitors will stuff their keywords to get to the top you'll have to do it too. It would become an arms race for who can stuff the most garbage into their indexes to "win". It just doesn't work.

All search engines that attempt to be useful will have to filter out the junk. You just have to trust that the search engine you are using isn't withholding results from you that it considers "bad" (eg: "misinformation" (i.e. stuff somebody disagrees with)).

And to me, that is the crux of the debate really. Nobody wants spam for search results--everybody agrees with that and there is no real debate about filtering that crap out. The argument really is should a very large company that has a huge market share get to decide what constitutes "fact" and what is "misinformation". Based on 2.5 years of experience so far, what was once deemed "misinformation" has a sneaky way of becoming "factual information". Labeling and hiding "misinformation" because it goes against some narrative pushed by incredibly powerful entities is very scary and there was a hell of a lot of exactly that going on during this covid crap.

I used to fall on the side of "private companies can do whatever they want" but now I'm not so sure. Companies like FB, Twitter or Google play a huge role in shaping politics and society. I'm no longer convinced it is okay to let them play the role of "fact checker" or anything like that. Filtering spam is one thing, but hiding "misinformation" is entirely different.

Your last point is also the one (aside from the economics) I am the most interested in.

I think we live in a world now where we are so used to a few tech giants mediating everything for us that we can't even imagine other solutions to this problem, but it's also how we got to this point in the first place.

>You simply can't trust sites, who have a strong incentive to get to the top of SEO rankings

Why is it not enough to punish sites that abuse the keywords?

Who is the one who punishes the abusers? How can you scale the solution to deal with billions of pages?

The users punish.

You need a trustworthy core by which you can judge the vote of new users. You can incorporate them until somebody complains about a result that is out of place.

This doesn't have to fully scale. There are many pages without monetary value that won't be manipulated. The tags are an additional signal that can be used where they work. If they don't work, they can be ignored.

But it will scale because there are far more consumers than producers.

People would abuse that for SEO purposes within seconds.

The market need would then be shifted to the best search interfaces instead of who has the most money to build the biggest index. A much better focus, IMO.

I believe that is precisely what the project is aiming to do, and to turn it into a public resource.

I’d rather see them publish a federated search of their own content.

Your comment prompted me to check out Searchcode, looks very interesting. How would the federated search model work in this example? Instead of you having to index the various code repositories, they would index themselves and make their search of those indexes available via a federated API?