So, you wear gloves at the grocery store, then you touch your phone with them? Wearing gloves isn't going to help unless you also wash everything you touched with gloves.

If you're concerned about touching things when out wash your hands when you get home (and anything else you've touched while out).