Show HN: Turn any website into a knowledge base for LLMs

Re:Show HN: Turn any website into a knowledge base for LLMs by tompec on Wednesday July 31, 2024 @05:43PM (#41123782)

It does respect robots.txt when crawling. I'll add more details about this in the docs.

[ Parent | on HN ]

Re:Show HN: Turn any website into a knowledge base for LLMs by pryelluw on Wednesday July 31, 2024 @06:53PM (#41124393)

I appreciate the reply. As someone who runs multiple CMSs it’s painful to deal with the ai crawlers these days. Specially the ones that don’t respect my terms.

[ Parent | on HN ]