You should ensure that they are somewhere under your organization's direct control and you can do a build without going out to the Internet. It's incredibly dangerous to have your build process depend upon random code pulled from the Internet, as left-pad showed.

Whether that means checking it into your VCS depends on which VCS you use. In general, Git does not handle large binaries well. Check out git-annex though, or use a package manager that lets you setup a local mirror.