I have known quite a number of people who have taken modafinil, and none of them experienced more than minor side effects. I have several friends who stopped using it for long periods because they built a tolerance (tolerance != addiction), but none experienced withdrawal symptoms.

I used adrafinil for a while because it has the advantage of being unscheduled in the US, and I noticed weak tolerance building, but the only side effects I had were insomnia (while it was in effect) and mild headaches which went away with aspirin. The only reason I stopped taking it was that they revoked its approval in France, and none of the generics from other countries that I've found seem to work.