Re:A look at search engines with their own indexes (2021)
by marginalia_nu on Tuesday June 21, 2022 @03:51PM
(#31828335)
The problem with this is that you get significant benefits from a larger crawling corpus, even if you index just a small portion of it, the rest will inform that portion.
That's a real problem that is ultimately hard to get around. Like my index is fairly small as it is, but even so it requires more hardware than say a student could afford. Like it's not enough that I'm dependent on external funding, but it's still not something you slap together for fun and then get bored of.
That's a real problem that is ultimately hard to get around. Like my index is fairly small as it is, but even so it requires more hardware than say a student could afford. Like it's not enough that I'm dependent on external funding, but it's still not something you slap together for fun and then get bored of.