Not true. A quick hand wash gets rid of almost all the bacteria. Fungi take a bit more scrubbing to get rid of.
I found this out in first year university biology. We swabbed and cultured our hands before and after washing with plain soap. Almost no colony forming units of bacteria are left after a short wash. A reasonable amount of the fungi is still there though.
Recently I read a study (or talked with a doctor who told me about it, can't remember) saying that it is actually more efficient to use disinfectants occasionally (ie on demand) instead of washing one's hands all the time. People tend to be wary of disinfectants because they tend to sting on the skin, but apparently the stinging is only a sign that the skin is already damaged. Apparently the disinfectants also contain stuff that is good for the skin, so using them is not as bad as it seems.