News for hackers. Stuff that matters.
Re:Indexing Code at Scale with Glean by jepler on Wednesday January 01, 2025 @06:44PM (#42570261)
My mind just balks at the idea of having so much source that a 2020s computer could take hours to index it. ctags is nothing special (both in terms of optimization but also the level of detail it gets to: just global function identifiers) and looks like it runs at about 400MB/s on a single core of an i5-1235U. But still it looks ctags could process about 100TB in 4 hours across 16 threads on a workstation class CPU...
Re:Indexing Code at Scale with Glean by DylanSp on Wednesday January 01, 2025 @06:52PM (#42570300)
It sounds like the indexing time/complexity is increased a lot by the amount of detailed data they're storing. They mention determining which `using` statement is used to resolve each symbol reference in C++ source, to enable dead code detection; that's going to require some sophisticated analysis.
Re:Indexing Code at Scale with Glean by menaerus on Thursday January 02, 2025 @03:02AM (#42572637)
Correct, you need to build an AST representation of the code that you want to index. Essentially, it's a compiler frontend pass and which is why it takes so much longer than what ctags heuristics do. Now think millions of lines of code, multiple build configurations, the amount of RAM you need, etc. Multiple branches, or even smaller revisions/commits, is also a big computation problem.

That said, Glean seems to be reusing the indexer from LLVM/clang for C and C++.

> The C++ indexer ("the clang indexer") is a wrapper over clang. The clang indexer is a drop in replacement for the C++ compiler that emits Glean facts instead of code. The wrapper is linked against libclang and libllvm.

[1] https://glean.software/docs/indexer/cxx

Re:Indexing Code at Scale with Glean by phyrex on Wednesday January 01, 2025 @08:32PM (#42570837)
It's a mono repo across a dozen languages (good luck with ctags) that tens of thousands of developers commit to every day. Even if you'd spend the hours indexing it locally, it would be out of date right away.
Re:Indexing Code at Scale with Glean by UltraSane on Wednesday January 01, 2025 @08:52PM (#42570922)
The whole point of indexing data is to perform very expensive computation once and leverage the result many many times and it works really well.
Re:Indexing Code at Scale with Glean by kllrnohj on Wednesday January 01, 2025 @09:09PM (#42570987)
You kinda said it yourself already - ctags is fast because it's producing almost nothing of value. Being fast at doing nothing isn't impressive.

Try doing the same with C++ and more indexing options enabled, such as with something like universal-ctags, and a larger code base, say Android's repository aught to do it. Are you still getting 400MB/s? Nope.