I also want pdfs and other files indexed
Typesense operates on JSON documents. So you'd have to first extract the data from PDFs / office docs into JSON and then index it.
There are libraries that let you do the conversion:
There are libraries that let you do the conversion:
A crawler is probably not the best tool for this if you use some form of site generation, as you can also use it to generate the information that goes into Typesense.