I work with both. You will not have any problem working with OS X. Some APIs incredibly similar in most kind of applications. I would even calling 'learning' a bit of a stretch. You will pretty much just acknowledge, not learn (in a broader meaning of the word).

Career wise, It will matter only if who is hiring you don't understand how similar iOS an OS X development really is.