News for hackers. Stuff that matters.

A comment on Ask HN: Strategies or tools for embedding multiple file types?.

Re:Ask HN: Strategies or tools for embedding multiple file t... by skeptrune on Wednesday April 09, 2025 @02:55PM (#43636071)
Strongly recommend using Apache Tika[1] for this. It's industry standard for ubiquitous document text extraction.

You can take the text output from Tika, chunk it with something like Chonkie[2], and embed it for your search index.

-[1]https://tika.apache.org/

-[2]https://chonkie.ai/

[ Parent | on HN ]
Re:Ask HN: Strategies or tools for embedding multiple file t... by spruce_tips on Wednesday April 09, 2025 @02:56PM (#43636092)
sweet this looks great. hadnt heard of chonkie but tika was on my list. thanks!
[ Parent | on HN ]