I may be misunderstanding this but if my users download a pixel/whatever from my site not from google analytics, then google does not know anything but my server IP

Or does client side mean the user downloads JS which does some investigation and reports back?