I average colors over the past X number of frames (where X is configurable). This might be a personal preference, but the sense of immersion is lost with rapid changes in colors or brightness. That's why I do this kind of blending.