To look on the bright side, my boss was amazing, a truly unique individual whom I thought so much of, I asked to be the godfather of my child.
Despite the chaos of the IT team, almost everyone I worked with was great. Everyone I met was really good at something, but I learned a lot from the people around me and had the opportunity to mentor a young engineer who ended up far exceeded me, which was rewarding in itself. I have several life-long friends from that company. People who have been with me through a messy divorce. A particular friend, when my wife left, called or texted every single day for six months-- just out of the goodness of her heart. The text was always the same, "Hey, wanna get some coffee? How are you doing today?" There are beautiful people working in even the most toxic places.
When I wasn't learning about engineering, I was learning about optics, about politics, and about the real world, and once again, I was winning, which was intoxicating.
So why ultimately did I stay? Because it was comfortable being the smartest person in the room. Because it was fun having latitude enough to start my own department? Because it was fun being engaged in this real-life game and winning? Because I'm crazy?
Things got far nastier than I went into. Once the IT department was openly trying to get me fired, things got personal. Really personal. It would have been easier to move on, but, I hate being bullied. Allow me to repeat myself, I HATE being bullied, and that's what this was in the end, organized harassment.
Through a set of unusual circumstances, I ended up befriending the main IT guys wife-- the man who decided he could play with my life for no reason at all, and his wife had a crush on me. So I and slept with her, and I am still sleeping with her four years later. To quote that great boss of mine, "Revenge is a dish best served every day."
Also it's not always so easy to up and quit every time someone looks at you the wrong way.
Deciding to leave doesn't mean put in your notice. It means look elsewhere. Leaving a job without another secured is always precarious absent a nest egg.
And I mean incompetent-- our network would go do down for hours, every day, and the problem lasted almost a year. It got so bad our "source control" became yelling "Whose got the latest main.cs?!" and walking it over with a USB key.
I realized we needed to disintermediate the IT group, and my boss was supportive. I ended up establishing our own, parallel IT department with a business cable line, some routers, and NAS devices. I outsourced any task to any cloud service (this was over a decade or so ago when that was somewhat unusual) — eventually hiring our own IT person even. People would ask for access to our network because it was more reliable :-)
Then things got vicious. They actively interfered with our group, trying to get us fired (they had some success with this previously). They were able to get our IT guy fired. Things just went on that way for years. They'd find some way to make trouble, and then we'd route around it. Once they even took my computer for two days, denied they had it, then returned it but locked me out of it. I was certain this was part of some scheme to get me fired, so I turned the computer off, zeroed the hard drive, and req'd a MacBook to work on.
They all eventually got fired when someone took note of their gross incompetence. One of their replacements was eventually fired for embezzling, proving that despite thinking we had the worst IT department anyone could imagine-- we really didn't.