I think you are thinking about this the wrong way. nilpotence doesn't relate to smoothness (of what) and differentiability (of what). Dual numbers happen to, the way I see it, model forward-mode differentiation. There are a couple of ways to see how eps^2=0 is a good model. But the easiest to get across is that in many calculus derivations, you take dx^2 to be 0, since dx is "a very small number".