I would generate a JSON feed for the site and upload that (I am curious if this can detect duplicate records. Edit: it can using the UPSURT call), that way you could even have full text search.

A crawler is probably not the best tool for this if you use some form of site generation, as you can also use it to generate the information that goes into Typesense.

I also want pdfs and other files indexed

Typesense operates on JSON documents. So you'd have to first extract the data from PDFs / office docs into JSON and then index it.

There are libraries that let you do the conversion:

https://github.com/modesty/pdf2json

https://github.com/microsoft/Simplify-Docx

You can index your crawl data with StorkSearch[0] then use its js interface for search.

[0]https://stork-search.net/