The motivation was many faceted, but the basic impetus for most people in the server mod world is that the vanilla server was so inadequate for running a server.
I got involved with a small server, and (quickly! that was a mistake haha) started to help sysadmin the box. I got us on to hMod, which was terrible in many ways but provided a vision for the future - a central, modded server, with plugin points that proffered server admins incredible flexibility configuring their servers.
hMod was run by a single (apparently inexperienced) dev, and the community grew too quickly for him so bukkit was formed.
By this point I had started contributing to hMod, but realised it was a sinking ship and switched my efforts to bukkit. It started as trying to build plugins for our server, but as I started getting commits into the core project it quickly ramped up into full on team participation.
The dev team was really really fun to be a part of. I was on mumble and irc all day talking to the other devs, and would often spend over 8 hours working on code or talking about plans. Upgrades were hectic, and everyone pulled long shifts when they happened.
As time went on and the community grew it became more like a real job, and momentum kept me going for a long time. Eventually I actually got a real job, and my participation waned.
I can't speak for others, but my motivation seemed to be a spectrum that started with scratching an itch end ended with the satisfaction of contributing to a community; the knowledge that your code is being used by thousands, if not millions, of people.
What's the motivation for running a server? and how do people using the game decide which server to use and why? I'm really stuck at the basics.
Larger servers have complex mods that add tremendous functionality to the game. Some may have roleplaying mods, where you can find a role (blacksmith, lumberjack, etc) and participate in the economy. Others may set up a world just for pure creation, a way to make sculptures.
There's a lot Minecraft has been extended to, so there are many reasons you may have to set up a server.
You can either join a server someone else is running, or you can set up your own server.
The Minecraft software had severe limitations around things like "griefing protection" (preventing people causing havoc); charging for stuff (which is one way servers used to raise funds); or minigames.
Looking back, I suppose it was a bit odd that there was a group of people, most of them quite a bit older than me, who spent large amounts of time working on this project.
From the bits and pieces of personal information that I gathered about my team members, many of them, including the team leader, had 'proper jobs'. One of them actually worked as a lead level designer on a triple-A game.
I got the impression that they spent all this time on the modification because they loved building games, they loved not being constrained by budgets or non-passion related factors, they wanted to build up a portfolio to find a way into the gaming industry, or 'all of the above'. And it worked for some of them.
I'm not sure if this is the case for a majority of 'modders', but I suspect the reasons I mentioned are quite common. If you love developing things, a game is a really great thing to build. It's, ultimately, purely about fun and joy. It doesn't have to be useful. That might be enough for those who work on (boring) useful stuff during the day.
Which, of course, is why modders should be encouraged to use platforms where you retain greater control of the project, which is why I'm a fan of OSS gaming.
People can and do build things without dollar signs in their eyes.