Re:Show HN: Cognita – open-source RAG framework for modular ...
by ComputerGuru on Sunday April 28, 2024 @11:38AM
(#40189399)
Does a "web" data source only scrape the individual page or linked pages as well? I'm assuming the former. What would be the least painful way to ingest a knowledgebase (say a wiki-like site) from the web?
[ Parent
| on HN ]
Re:Show HN: Cognita – open-source RAG framework for modular ...
by s1lv3rj1nx on Sunday April 28, 2024 @02:26PM
(#40190623)
It can scrape linked pages too by defining the depth but make sure the depth parameter is not too much else it will consume too much memory and time.
Re:Show HN: Cognita – open-source RAG framework for modular ...
by ComputerGuru on Monday April 29, 2024 @05:23PM
(#40204298)
Playing around with the UI, I cannot see where that depth would be set. Is it not a per-datasource variable?
Is the "scrape linked pages" configured to be "sandboxed" within a url hierarchy (so adding example.com/foo/ would add all linked pages that are also under example.com/foo/) or not (so it would also include linked pages to other domains or subfolders)?