Assuming you rinse them off and you have a reasonably large amount of cutlery, I'd say you are definitely too paranoid about this specific thing. Your entire kitchen is a bacterial growth medium - just leaving things out means they get covered in bacteria. Those bacteria will grow and this is fine - the most efficient use of your thoughts is fixing the really large offenders, one of which is: Do you properly wash your hands (most people do not, or at all)?

Wash them quickly. Bacteria growth is not significant on the order of minutes.

It's not the growth during washing I'm worried about, but that not rinsing them will leave them covered in a growth medium that will significantly accelerate their growth over the next hours or days.

I am pretty sure that the amount of bacteria will be lesser than what we carry.

But yeah at times I have the same thing running on my head.

I was able to reduce my water bills by almost 30% just by manually cleaning the utensils and reducing the washing machine usage.