All they need to do is see what wifi connections I can reach on my phone. In any urban area you can easily reach 50+ wifi connections standing in one spot. If google knows X wifi connection corresponds to Y range of a physical location because of someone else left their location and wifi on, then they can tell that if I can reach 4 or 5 different wifi connections at once then I must be at a certain address.

It's like browser fingerprinting. It doesn't take very many bits of entropy before you are identifiable