Show HN: Turn any website into a knowledge base for LLMs

Re:Show HN: Turn any website into a knowledge base for LLMs by pryelluw on Wednesday July 31, 2024 @02:52PM (#41122186)

Does this respect robots.txt?

[ Parent | on HN ]

Re:Show HN: Turn any website into a knowledge base for LLMs by danirod on Wednesday July 31, 2024 @04:12PM (#41122961)

[delayed]

[ Parent | on HN ]

Re:Show HN: Turn any website into a knowledge base for LLMs by srameshc on Wednesday July 31, 2024 @03:54PM (#41122836)

Valid question and I am sure it doesn't.

[ Parent | on HN ]

Re:Show HN: Turn any website into a knowledge base for LLMs by tompec on Wednesday July 31, 2024 @05:43PM (#41123782)

It does respect robots.txt when crawling. I'll add more details about this in the docs.

[ Parent | on HN ]

Re:Show HN: Turn any website into a knowledge base for LLMs by pryelluw on Wednesday July 31, 2024 @06:53PM (#41124393)

I appreciate the reply. As someone who runs multiple CMSs it’s painful to deal with the ai crawlers these days. Specially the ones that don’t respect my terms.

[ Parent | on HN ]