> And this is the crux of where we disagree. This is a messy part of reality which should be, as far as possible, abstracted away from the algorithms which need to operate on the data presented to them. If I'm running a Kalman filter I don't want to have to design in my filter around frequent gyroscope dropouts because image captures are happening, I want my system to have guaranteed behaviour that this won't happen.

Annoyingly in a lot of real world setups you can't have these guarantees. Your gyroscope, camera, etc are all producing data asynchronously often with slightly different clocks, and they all have different little idiosyncrasies and failure modes.

For example the Mars helicopter almost crashed because it missed a single frame. https://mars.nasa.gov/technology/helicopter/status/305/survi... If possible you absolutely want to fix the frame drop in the first place, but your algorithm should also be able to handle the drop out (or at least reset/recover).