Not to be too dismissive about your experiences, but making medical-related decisions based on anecdotal evidence is a really bad idea -- the debate over vaccines is a good example of this IMO.
Have a look at reenw's numerous comments in this thread for references to sources/studies that suggest that it can be that easy...
Tap water and mis-care of contact lenses induces significantly elevated risks of eye infection. Your lottery ticket comparison is apt, since as the article and some other comments here reflect, you are gambling your sight with this attitude. Same one everyone hears from the guy who doesn't wear glasses while he's running a lathe. The difference being, at least that guy can argue his skill reduces his risk of injury.
Would it be wise to stop looking both ways before crossing the street because the first three times you tried it you weren't struck by a car?
All that said, after reading a number of different resources, I am considering such a change.
A shame, preservative free saline is expensive.
It's not the lottery, that's how infections work.
Maybe I'm just that lucky. A shame my lottery tickets disagree.