Doesn't it depend on whether there's a stress/emphasis on the verb? I'd to seems wrong because the had in I had to is stressed, so the had just seems missing. In I'd love to - the would isn't stressed. If it is, like "I would love to, but I can't" then the contraction isn't used. It's how contractions arose in the first place, I guess.

I don't know the "real" rule for when you can contract and when you can't, but I'm sure that there's some linguist somewhere who has written a paper on it. The reason that I didn't try to explain the stress that native speakers feel by talking about modal verbs is because I don't think that you can say in general that you never contract a modal verb. For example you can say "I would talk to Jim" or "I'd talk to Jim" where is "would" a modal verb modifying "talk". Maybe it's the case that the only time you have a phrase in the form of [auxiliary verb] [unconjugated verb] is when the auxiliary verb is being used as a modal verb? I don't know. I don't have any expertise in linguistics, I just know the terms because I took a class on it in college.