You cannot know where a server is physically located, as even self-reported locations are incorrect (by accident and/or on purpose) and there is no such thing as non-3rd party data about physical location as servers as again, it's self-reported and any registry can say whatever they want in regards to where a server is located.

There are many different 3rd party sources for geo locating IP addresses, maybe MaxMinds is the most popular one. But if you acquire a few, you'll notice that sometimes even they don't agree where the location of a server is.