Re:Two Upstart Search Engines Are Teaming Up to Take on Google
by stephantul on Tuesday November 12, 2024 @10:06AM
(#42115997)
I meant this in response to the parent that Common Crawl only updates every month, which seemed to imply that this was sufficient.
This is too slow for a lot of the purposes people tend to use search engines for. I agree that you don't need to crawl everything every minute. My previous employer also crawled a large portion of the internet every month, but most of it didn't update between crawls.
Re:Two Upstart Search Engines Are Teaming Up to Take on Google
by arnaudsm on Tuesday November 12, 2024 @09:10AM
(#42115587)
Big fan of your work Viktor, thanks for everything you build and how much you document it
Re:Two Upstart Search Engines Are Teaming Up to Take on Google
by tech234a on Tuesday November 12, 2024 @08:55PM
(#42122130)
See also: IndexNow [1], a protocol used by Bing, Naver, Yandex, Seznam, and Yep where sites can ping one of these search engines when a page is updated and all others will be immediately notified. Unfortunately it does seem somewhat closed as to requirements for joining as a search engine.
Re:Two Upstart Search Engines Are Teaming Up to Take on Google
by mdaniel on Wednesday November 13, 2024 @11:47AM
(#42127631)
What year was this thing created, because the /.well-known URI scheme[1] has existed for a long time and is designed for this kind of junk https://www.indexnow.org/documentation#:~:text=Hosting%20a%2...
1: https://www.iana.org/assignments/well-known-uris/well-known-...
Re:Two Upstart Search Engines Are Teaming Up to Take on Google
by tech234a on Wednesday November 13, 2024 @09:05PM
(#42132418)
It was announced October 18, 2021: https://blogs.bing.com/webmaster/october-2021/IndexNow-Insta...
Re:Two Upstart Search Engines Are Teaming Up to Take on Google
by mdaniel on Wednesday November 13, 2024 @11:57PM
(#42133275)
The irony of a website from 2 major search engines looking like it was made in the early 2000s doesn't escape me. But, to my original point, there's absolutely no way they were ignorant of well-known URIs
The problem of being able to provide fresh results is best solved by having different tiers of indices, one for frequently updating content, and one for slowly updating content with a weekly or monthly cadence.
You can get a long way by driving he frequently updating index via RSS feeds and social media firehoses to provide singnals for when to fetch new URLs.