"and even a git status takes 30 seconds to run"

That's a problem with your vcs, not with the (defensive) practice of tracking external dependencies.