News for hackers. Stuff that matters.
Re:Ask HN: Has anybody built search on top of Anna's Archive? by showerst on Wednesday June 04, 2025 @08:59AM (#44180291)
How would that scale to 10TB+ of plain text though? Presumably the indexes would be many gigabytes, especially with full text search.
Re:Ask HN: Has anybody built search on top of Anna's Archive? by wolfgang42 on Wednesday June 04, 2025 @03:34PM (#44184632)
The client only needs to get indexes for the specific search; if the index is just a list of TF-IDF term scores per document (which gets you a very reasonable start on search relevance) some extremely back-of-the-envelope math leads me to guess at an upper bound in the low tens of megabytes per (non-stopword) term, which seems doable for a client to download on demand.