Re:Show HN: Doctor – tool to crawl and index websites and MC...
by kixpanganiban on Monday May 12, 2025 @09:24PM
(#43968930)
Hi! I wrote Doctor because I keep struggling with grounding on docs when working with agentic code editing (ex Roo, Claude Code).
Doctor uses crawl4ai to crawl websites, and then chunks and embeds them with langchain + litellm + openai, and finally stores all the vectors in duckdb. This allows your LLM to query the docs using semantic search over MCP, giving it grounded and up-to-date information for the things you're working on.
It requires an OpenAI key for the embedding process, but I'm working on giving users options in the future (different providers, local embedding using something like DPR or other transformer libs, etc.)
Doctor uses crawl4ai to crawl websites, and then chunks and embeds them with langchain + litellm + openai, and finally stores all the vectors in duckdb. This allows your LLM to query the docs using semantic search over MCP, giving it grounded and up-to-date information for the things you're working on.
It requires an OpenAI key for the embedding process, but I'm working on giving users options in the future (different providers, local embedding using something like DPR or other transformer libs, etc.)