I think Yandex have their own index and some others too like Marginalia, but the latter couldn't be called "in the same category" as the other three.

Yandex also suffered a security breach and their source code is available[0] although utilizing it in any way is ethically and legally dubious (at best).

[0] - https://arstechnica.com/information-technology/2023/01/massi...

Brave Search has its own independent index too - https://brave.com/brave-search-beta/

> Pretty sure that only Google and Microsoft have the money and resources to crawl the entire internet. Or perhaps the only that can AND are willing to.

Money and resources and a dominant-enough position so that your crawlers are not blocked by websites.

Unfortunately.

There's definitely gatekeeping on websites but having done a bunch of crawler work I can say you'd be surprised how rarely a site will outright block you if you just do the right things: have an identifiable user agent with a working URL that explains what your crawler does, respect robots.txt, implement polite crawling. As for actually being able to crawl the whole thing though, yeah it's stupidly expensive :-/

Mwmbl has its own index but it's orders of magnitude smaller than commercial search engines.

What about Common Crawl?

https://commoncrawl.org/