It's an easy mistake to make. Incidentally, "infer", for some reason, is a word I find unusually pleasing.

EDIT: as a non-native speaker I distinctly remember first reading the word and happily using it ever since, when appropriate.

And a similar possibly wrong usage is "conference" when used as a verb, maybe, since "confer" is probably okay. But "conference" seems to have become accepted, I've heard it a bit. There's this trend (from many years, not just recent) to make nouns into verbs too, e.g. "Skype me". The "confer" form as a verb probably sounds archaic to many, since it's not commonly heard or read (AFAIK).

Perfectly fair — I'm sure you're not the first to inference something. And yes, 'infer' is an interesting word, especially when you bring in Latin.