You may be thinking of FaceTime. Apple was going to open that, but a successful lawsuit by a patent troll (I think it was VirnetX) put a stop to that, because Apple had to change FaceTime to proxy through their own servers, which meant that opening it up wouldn't do any good.

That said, VirnetX has also sued Apple about iMessage, so it's possible there's something to what you've said (but I've never heard it said before that Apple was going to open up iMessage).