Given that the goal is only a small subsection of topics, might scalability not be as critical to the success of the project? If the engine is specifically crawling official programming language documentation, help forums like StackOverflow, and select sites/pages (think university professor with resources about how to implement xyz), might that be enough to not necessitate scaling even farther?