The problem is when your system clock is so far out of whack (and/or unpredictable) that even ntpd won't help you. Get enough supposedly-identical machines running and this will probably bite you too.
tinker panic 0
Ironically, often a misconfigured NTP server screws up GPS fixes (it somehow seems to be necessary for assisted GPS, which relies on the cooperation of the mobile network or Internet to faster acquire a GPS fix).
http://stackoverflow.com/questions/8308412/gps-how-ntp-time-...
Genuinely curious, is it actually true that you can get sub-100ns time accuracy on a mobile phone? Is this because you have a direct connection with the satellite, which is calculating where you are? I would have assumed that the uncertainty on the latency of any such connection would be higher than 100 ns.
As long as you have access to at least 4 satellites, triangulation (quadri-angulation?) allows you to determine your precise location in all 4 dimensions, three of space and one of time.
GPS time is theoretically accurate to about 14 nanoseconds. However, most receivers lose accuracy in the interpretation of the signals and are only accurate to 100 nanoseconds.
Mobile phones use CDMA stuff to sync their clocks, not GPS
NITZ is terrible. You only get one-second accuracy, a GMT offset, and a DST flag, and they can all be wrong at the whim of someone at the phone company.
Is there an expansion card or USB device that can get the time from GPS and feed it to something like ntpd (and adjust for leap seconds from data downloded from other networks)? Or would that be impractical because of all the walls and metal cages that surround the typical server? What about an antenna on the roof of the datacenter that supplies GPS time to every server in the building?