Close the git perspective? Right-click project, team -> disconnect perhaps?

Honestly, I'm curious. It's never occurred to me that having it there, even if you don't use it, could be problematic. What is it touching that bothers you so much? It's not like it's going to jump in your folder and start committing stuff at random.