I do something very similar, but use two time constants. One to determine how many consecutive samples are needed to change from the established state, and a second one, as suggested here, to ignore subsequent input for a bit after a registered state change to allow for bouncing before going back to the receptive state. This is a bit more robust, as practical systems are always at the mercy of occasional glitches in electromagnetically noisy environments.

With well-chosen constants, I've found similar behaviour to the above; low latency and high performance at once.