News for hackers. Stuff that matters.
Re:Show HN: Cognita – open-source RAG framework for modular ... by ComputerGuru on Sunday April 28, 2024 @11:38AM (#40189399)
Does a "web" data source only scrape the individual page or linked pages as well? I'm assuming the former. What would be the least painful way to ingest a knowledgebase (say a wiki-like site) from the web?
Re:Show HN: Cognita – open-source RAG framework for modular ... by s1lv3rj1nx on Sunday April 28, 2024 @02:26PM (#40190623)
It can scrape linked pages too by defining the depth but make sure the depth parameter is not too much else it will consume too much memory and time.
Re:Show HN: Cognita – open-source RAG framework for modular ... by ComputerGuru on Monday April 29, 2024 @05:23PM (#40204298)
Playing around with the UI, I cannot see where that depth would be set. Is it not a per-datasource variable?

Is the "scrape linked pages" configured to be "sandboxed" within a url hierarchy (so adding example.com/foo/ would add all linked pages that are also under example.com/foo/) or not (so it would also include linked pages to other domains or subfolders)?