Unless you mean just using the mod as a key to access the server in which case you'd be better off just charging for passwords to the server because otherwise you have to worry about people using pirate copies of your mod to connect.
I'm guessing the pirate effect wouldn't be an issue because the "pro mod writers" would be about 10x more excited about the topic than "pro server operators" for obvious reasons so the mod store writers wouldn't do anything too dumb... I hope. One rather trivial and easy to implement solution is MC currently has a centralized auth system for MC users that existing servers talk to (I have a server in my basement, and I am made well aware from my kids when the MC auth servers are down), and extending that a tiny bit to auth both MC and mods isn't much work other than a bit more traffic to the auth server, which they can probably afford by skimming off mod sale prices. So you can tell the server all day that you have a copy of mod name jiggy-2011, but when the server turns around and asks the existing centralized MC server if username jiggy has actually paid for mod name jiggy-2011, then ...
If your server requires 1.6.4 and railcraft 1.2.3.4.5.6 well then anyone who connects requires MC 1.6.4 (free, no problem) and ALSO requires railcraft 1.2.3.4.5.6 and mods are stereotypically hosted on the scummiest pr0n-iest adfest drive by download sewers of the internet (sorry, but its true)
So say you want to host a server called "jiggy's minecraft". Simply require a mod to log into your server called jiggy-2014 for the entire year of 2014 and sell it in the in app store for "annual fee". Maybe drop the price by 1 / 12 every month thru the year. Guess the mod name you'll require in 2015, yes your mod will be called jiggy-2015 and be priced at your annual fee.
The mod might even provide something useful to the players. Maybe. This will probably kill/damage the ecosystem, maybe you'll never get to play railcraft, TE, and buildcraft on the same server ever again. That would suck. It is possible "real mods" won't be balkanized and only "subscription mods" will be balkanized.