Re:Ask HN: I want to dive into how to make search engines
by oikawa_tooru on Thursday August 25, 2022 @02:24PM
(#32597875)
Hi thank you for your answer. Actually my original idea was to build it for something else rather than the web in particular. Can you please advice me on which masters I should pick between distributed systems , machine learning and theoretical computer science/algorithm if my end goal is this?
Re:Ask HN: I want to dive into how to make search engines
by matt_s on Thursday August 25, 2022 @05:01PM
(#32599894)
I don't think any specific MS is going to cover everything for building a search engine. You may want to look at people that work on Lucene [0], for example, and connect to see what their background might be. My guess is there is much more algorithms (types of searches and their big-O tradeoffs, sorts, etc.) involved than dist systems/ml but I don't work on search engines. Its open source so you could just download it and tinker. You may find that a degree isn't necessary to build one.
I think a major effort, if its not web, is defining a common data structure to represent the varying source data sets.
Edit to add: OP if you want to study this, start smaller than "search the web" and get the major component concepts down. Expanding to the web will involve distributed systems stuff to scale but there's only like a handful of companies doing that yet there are many more companies that need a search engine on in-house data.