It's not a huge pain, but it's a prominent feature of developing for devices that are frequently updated.
We always use the latest XCode and the latest SDK. Often there are language features that greatly reduce development time, like the NSDictionary/NSArray literals for example. The only downside is that later XCode versions don't support low-iOS version simulators, so you have to test on-device only when supporting older devices. But this all helps justify to users when you stop supporting older devices.
Most of the new features in iOS 6 that would be used by libraries are developer conveniences and can be ignored with a bit more effort. (e.g. Auto Layout, UICollectionView)
As far as different versions of Xcode, there's really no reason for developers not to use the latest in my experience, so targeting that is likely fine.
Building for iOS in general has improved a lot. Apple has really cleaned up iOS and Xcode in the last two years, with ARC being a huge improvement for developers. Seeing that iOS 6+ is now 97.7% of devices (https://mixpanel.com/trends/#report/ios_7), we feel that focusing on iOS 6+ is a safe bet going forward.
http://appleinsider.com/articles/13/12/05/apple-pegs-ios-7-d...
I have a somewhat related question for anyone who works on tools/frameworks (for iOS I guess)... is it a pain to deal with iOS versions, ARC/non-ARC, and anything else? I was reading the quick start guide here and I noticed it said iOS 6.0+, so I guess you just have to look at the market share and make a trade off as to how much you want to support?
I've personally only worked on apps themselves, but I've always wondered if the people making Flurry, Tapjoy, etc have to go through hell getting their framework to work for every combo of Xcode and iOS out there lol.