Re:Ask HN: Has anybody built search on top of Anna's Archive?
by showerst on Wednesday June 04, 2025 @08:59AM
(#44180291)
How would that scale to 10TB+ of plain text though? Presumably the indexes would be many gigabytes, especially with full text search.
[ Parent
| on HN ]
Re:Ask HN: Has anybody built search on top of Anna's Archive?
by wolfgang42 on Wednesday June 04, 2025 @03:34PM
(#44184632)
The client only needs to get indexes for the specific search; if the index is just a list of TF-IDF term scores per document (which gets you a very reasonable start on search relevance) some extremely back-of-the-envelope math leads me to guess at an upper bound in the low tens of megabytes per (non-stopword) term, which seems doable for a client to download on demand.