https://en.wikipedia.org/wiki/Apache_Lucene
https://en.wikipedia.org/wiki/Apache_Solr
https://en.wikipedia.org/wiki/Elasticsearch
those who dont know history will repeat it, etc
then maybe the newer stuff like https://typesense.org/
to be clear i dont know these things but thats what I would do and I'd happily read fly.io style blogposts drip feeding out knowledge over time
I think a major effort, if its not web, is defining a common data structure to represent the varying source data sets.
Edit to add: OP if you want to study this, start smaller than "search the web" and get the major component concepts down. Expanding to the web will involve distributed systems stuff to scale but there's only like a handful of companies doing that yet there are many more companies that need a search engine on in-house data.
A different "information retrieval" perspective overall would be to take a deep dive into the career of Susan Dumais, http://susandumais.com/
You can decompose a "search engine" into multiple big components and figure out what you want to look at first:
(1) web crawler/spiders
(2) database cache of web content -- aka building the "search index"
(3) algorithm of scoring/weighing/ranking of pages -- e.g. "PageRank"
(4) query engine -- translating user inputs into returning the most "relevant" pages
Each technical topic is a sub-specialty and can be staffed by dedicated engineers. There are also more topics such as lexical analysis, distributed computing (for all 4 areas), etc.
If you're mainly focused on experimenting with programming another ranking algorithm, you can skip part (1) by leveraging the dataset from Common Crawl: https://index.commoncrawl.org/
Here are some videos about PageRank:
https://www.youtube.com/watch?v=JGQe4kiPnrU , https://www.youtube.com/watch?v=qxEkY8OScYY
... but keep in mind that the scope of those videos omits all of (1), (2), and (4).