Re:Creating an advanced search engine with PostgreSQL
by gomezjdaniel on Wednesday July 12, 2023 @05:54PM
(#36702213)
It is open sourced anywhere? I am curious
Re:Creating an advanced search engine with PostgreSQL
by paxys on Wednesday July 12, 2023 @08:29PM
(#36703555)
When building a search engine the biggest problem isn't QPS but the size of the data set you are indexing. If your search structures can fit in memory of a single machine then you can serve effective infinite QPS with ~zero latency. When it grows beyond that you have to get creative, and that's when all the different tradeoffs come into the picture.
Re:Creating an advanced search engine with PostgreSQL
by vlovich123 on Wednesday July 12, 2023 @08:30PM
(#36703566)
250k RPS seems like a lot when SQLite can’t do that today and I feel like inverted indices are more expensive (eg RocksDB gets to ~130k RPS but I think their numbers are on beefier hardware than my laptop or my machine isn’t set up well). Are you sure you wrote a general purpose database and those numbers are correct? I’m curious what you did to outperform these engines (I agree it’s possible, but I’m curious what techniques you employed)
Re:Creating an advanced search engine with PostgreSQL
by lionkor on Thursday July 13, 2023 @05:04AM
(#36706583)
I'd guess:
- everything was in RAM,
- it was written in a compiled language, and
- probably skipped worrying about too many crazy edge cases
It's actually very easy to make a fast basic database and search engine, even as an amateur programmer. As long as you understand basic CS algorithms and how to exploit the operating system and hardware, you can put one together in a month or two. Speed is not bad even with high-level languages; something like 250K QPS, back in 2003, on a laptop. Scalability isn't much of an issue either if you shard it. Indexing, locking, and consistency are more complicated than the storage and retrieval parts.
The big problem to overcome is the subjective nature of search. What do I really want to find? How do I find something when I don't know what I'm looking for? How do I get around people trying to game the system? How do I handle complex queries and datasets? That's when it gets orders of magnitude harder.