We chunk 2M+ files daily for our embeddings-based code search at Sweep, using a syntax tree-based chunker. We just published a technical blog post outlining this algorithm at https://docs.sweep.dev/blogs/chunking-2m-files.

At a high level, we run a similar algorithm to LangChain's recursive text splitter on a concrete syntax tree (CST) parsed using Tree Sitter. This ensures relevant contexts, such as function headers to their implementations, stay in the same chunk, ultimately producing high-quality search results.

For similar technical blog posts, check out our blogs at https://docs.sweep.dev/.