Re:Show HN: Cognita – open-source RAG framework for modular ...
by s1lv3rj1nx on Sunday April 28, 2024 @02:26PM
(#40190623)
It can scrape linked pages too by defining the depth but make sure the depth parameter is not too much else it will consume too much memory and time.
[ Parent
| on HN ]
Re:Show HN: Cognita – open-source RAG framework for modular ...
by ComputerGuru on Monday April 29, 2024 @05:23PM
(#40204298)
Playing around with the UI, I cannot see where that depth would be set. Is it not a per-datasource variable?
Is the "scrape linked pages" configured to be "sandboxed" within a url hierarchy (so adding example.com/foo/ would add all linked pages that are also under example.com/foo/) or not (so it would also include linked pages to other domains or subfolders)?