Depends on your threat modeling. Are you running untrusted code on the worker (maybe you have an org with thousands of engineers, maybe you're running builds from a public fork)? Sure, that's an issue. Are you a small startup? Take the convenience and focus on the bigger fish to fry.