I understand that what people typically mean is people who are “too political” where the pushing for interests stops being in service of some goal and starts to become its own point. But in trying to avoid the appearance of that, too many people fail to engage in any politics and in doing so actually cause more harm than good to what is important to them by missing the important point that it’s incumbent to speak up for things you need and believe in because people who allocate resources may have insufficient knowledge or insufficient time to dedicate to knowing your work that intimately.
It typically doesn’t stop being in service of some goal, but rather becomes entirely self-serving.
But when I say gravitate away from hard work towards playing politics it refers to something else of course. Getting others to do your work while still claiming credit is one example.
An engineer also has to eat lunch in order to function, but engineering is not the same thing as professional food critiquing.
---
To answer your question, it's pejorative because politics is about allocating resources one way or another, towards and away from particular social groups. It's inherently going to be making some of the people unhappy, 100% of the time. Real life politics also tends to not be done smartly and honestly at all times and in all places, and this can make some people upset as well.
Most people associate all of that together, and most people don't take an academic (or particularly analytical) approach to the subject.