If you use N sites which all adopt this authentication mechanism (i.e. widespread adoption); and if I can access your e-mail, then I can access all of those N sites. Furthermore, yes, because all those accesses look normal, nobody would detect it as unauthorized.