gets me thinking — this is totally doable for open source code. someone just needs to build some giant indexes across git repos + package repos (maven/npm/pip/etc)...
Then write plugins for VScode/intellij, and when developing you can right-click a symbol and “Find uses in open source code”
Does something like this already exist?
1. Vastly expanding the size of our global search index to cover every public repository on github.com, gitlab.com, and bitbucket.org. 2. Enabling LSIF indexing (https://lsif.dev) in every major language for compiler-accurate code navigation (go-to-def, find-refs) across repository/dependency boundaries.
The latter is already working on a subset of languages for private Sourcegraph instances, and we want to scale it to the entire open-source world. We think a single search box that covers all the visible code in the world and allows you to seamlessly walk the reference graph is super super powerful and someday will be a thing that most developers use every day.
Do OSS licenses distinguish between corporate usage of repository source as code vs as data.
That said, the community often takes authors' wishes into account.
Out of curiosity, why would you want to restrict this? It's not like Sourcegraph has any exclusivity to the concept or the ability to implement it.
But yeah, it could get expensive to keep the indexes current as things grow.
Sounds a lot like sourcegraph.
However, neither works great—maybe 30%-80% coverage?, and far from "in a way that gets near complete coverage" that the parent stated about the benefit of Google's monorepo.
It may be a case where you need 90 or 99 or even higher coverage before it becomes a game changer. But I think GitHub is headed there quickly.
For the latter, it's a known limitation in that we're only doing "fuzzy" or "c-tags-like" Code Nav — defs and refs are only matched by their textual unqualified symbol names. For some languages that's not a big deal, because symbol names tend to be unique. But for a language like Go, you might have dozens (or hundreds!) of methods named "Parse", and we don't currently distinguish those in our Jump to Def UI.
That's something we're actively working on, though! One benefit to the current fuzzy approach is that it's incremental (we only have to analyze changed files in a commit), and does not depend on tapping into a build or CI process. That makes it much faster — we have a long-running indexer service live and hot, and typically have new commits indexed and live in the UI within 1-5 seconds of receiving a push. And it requires no configuration on the part of the repo owner — no need to tell us how to build your project, or to configure an Actions workflow to generate the data. That makes it easier for us to support entire ecosystems when we roll out support for a new language. We've also tried to make it fairly easy for external communities to add support for their languages — this is all driven by the open-source tree-sitter parsing framework. This file, for instance, is where the symbol extraction rules for Go are defined: https://github.com/tree-sitter/tree-sitter-go/blob/master/qu...
We've been working on a new approach to generate more precise symbol data, while keeping those incremental and zero-config properties. It's not ready for public launch, yet, but it's close!
For those who want to learn more, I gave a talk at FOSDEM 2020 (back in the Before Times) going into more detail on some of these constraints and design decisions. (Though please note that this talk references a legacy way of defining symbol extraction rules, which relied on writing code in our open-source Semantic project.) https://dcreager.net/talks/2020-fosdem/
I don’t have any links saved, since they feel kinda sketch (ads all over the place). But it has been very helpful, unlike searching within GitHub for a string that I know is in the repo.
If the change is small or can be automated (i.e. changing # of parameters or function names), we run a script to make the change over the entire monorepo. This enormous CL is then approved by one of the Global owners.
If the change is complex (or non-obvious), you generally introduce the new API in one CL, change each use manually (say one CL per team), and then remove the old API in a final CL. In that case, you need each team to sign off. This isn't too hard in practice, teams are generally expected to approve cleanups.
We usually have a 3 steps process. 1st new API interface/service using the new dependency is added and runs in parallel with the existing one, 2nd we blast email to users of old one to migrate to the new one, 3rd when usage of the old one hits 0 it is removed. When a team slacks off in migrating (usually we have months between receiving the first notification and service shutdown) things escalate and a director might get involved.
Anyway, this means that migrating to the new interface is clients' responsibility.
However there's an exception: if the changes are trivial (like changing the the name of one of the API endpoints), we have an automated tool that basically performs this change across the whole codebase. These kinds of changes still need to be approved by the codebase owners and shouldn't break any test, so they must be really trivial. In this case is usually the team owning the API/Service that performs that.
For third party dependencies there are two main options.
1. Do it all at once. This is good for relatively small numbers of users or small API changes.
2. Introduce the new version, migrate users, then delete the old version.
Note that for third party libraries Google does maintain a fairly strict one version policy so 2 is always a temporary measure.
* Automation.
* Ability to change other people's code safely, possibly gated by code review approval. Cultural and technical (e.g. unit tests must exist and be maintained) barriers may apply.
* Making incremental changes that are backward compatible (at least until you migrated all clients so you can cleanup later)
I recommend this talk https://youtu.be/tISy7EJQPzI (even if the scope is broader than your question)
(ex-googler)
At BazelCon 2020, Borja Lorente from Twitter gave a talk [0] about how they were migrating away from Pants to Bazel. Seeing as Twitter was heavily involved in the development of Pants, not sure what that means with respect to adoption and evangalism once they complete their migration.
Lately it seems that Bazel has basically won in the space of open source projects using Bazel clones as more and more companies switch over.
One thing that is maybe not obvious: For an API author, code search in combination with a monorepo and the somewhat hermetic universe which is Google's code base provides immediate access to all uses of a library. You can see what worked well and what didn't, and it enables effective refactorings. That also means when writing a client, you can quickly figure out from other client code how stuff is supposed to be used. All this makes code search such an effective tool in Google's development environment (in the broader sense).
If anything like this were possible for open source (indexing code that depends on a library/API, across subrepos in any version control system, in a way that gets near complete coverage), it would enable similar possibilities of systematic improvement.
Alas, it does not seem realistic except in a few niches where the number of clients is bounded and code owners are willing and able to follow a protocol (approve changes to their code that unblock such global improvements.)