The jitter artifacts could be removed by applying some kind of temporal consistency constraints on the network predictions. But such an approach will also definitely introduce a lot of edge cases where a sudden change is actually ignored due to enforcement of the constraint. An adaptive consistency parameter could probably work better, but it's non-trivial to have a meta-algorithm to figure out the paramter.