Isn't it only a problem if they actually use it to identify you.

If they just use it the same way that google uses gmail I fail to see the problem.