But realistically this would mean I couldn't use an iPad with it's pen without connecting it to the internet first. I mean how would Apple bake every calibration for every display into all devices, even ones it hasn't even produced yet?

Because the screen was on the device when it was calibrated.

It's the display that needs calibration. Pen doesn't know where it is, the screen does. So each complete device or at least the screen needs to have a valid calibration table or a function for raw xy -> calibrated xy.