How on Earth do you react with reasonable latency if you're N milliseconds away from all sources of truth, and or on a residential-class Internet connection without all kinds of SLAs?

That's a security problem in and of itself.

Humans don’t react on those timescales. These systems where operators are doing things remotely have low latency things for any automation. It’s the “management” interface we’re talking about here.