News for hackers. Stuff that matters.

A comment on Ask HN: Has anybody built search on top of Anna's Archive?. Parent comment

Re:Ask HN: Has anybody built search on top of Anna's Archive? by ThatPlayer on Wednesday June 04, 2025 @05:33AM (#44178838)
I've done something similar with a static hosted site I'm working on. I opted to not reinvent the wheel, and just use WASM Sqlite in the browser. Sqlite already splits the database into fixed-size pages, so the driver using HTTP Range Requests can download only the required pages. Just have to make good indexes.

I can even use Sqlite's full-text search capabilities!

[ Parent | on HN ]
Re:Ask HN: Has anybody built search on top of Anna's Archive? by showerst on Wednesday June 04, 2025 @08:59AM (#44180291)
How would that scale to 10TB+ of plain text though? Presumably the indexes would be many gigabytes, especially with full text search.
[ Parent | on HN ]
Re:Ask HN: Has anybody built search on top of Anna's Archive? by wolfgang42 on Wednesday June 04, 2025 @03:34PM (#44184632)
The client only needs to get indexes for the specific search; if the index is just a list of TF-IDF term scores per document (which gets you a very reasonable start on search relevance) some extremely back-of-the-envelope math leads me to guess at an upper bound in the low tens of megabytes per (non-stopword) term, which seems doable for a client to download on demand.
[ Parent | on HN ]
Re:Ask HN: Has anybody built search on top of Anna's Archive? by qcic on Wednesday June 04, 2025 @10:16AM (#44180978)
Super interesting.
[ Parent | on HN ]
Re:Ask HN: Has anybody built search on top of Anna's Archive? by Aachen on Wednesday June 04, 2025 @10:46AM (#44181258)
I wonder if you could take this one step further and have opaque queries using homomorphic encryption on the index and then somehow extracting ranges around the document(s) you're interested in

Inspired by: "Show HN: Read Wikipedia privately using homomorphic encryption" https://news.ycombinator.com/item?id=31668814

[ Parent | on HN ]