Traits that are maladaptive are culled. Ones that have no effect are preserved, increased or diluted depending entirely on what other genes they appear with. Not the gene themselves.
If pale eyes were not beneficial in northern latitudes, we would still have blue eyes because the norsemen had them, and they were very successful. That gene will stick around until something forces it out of the pool because there is otherwise nothing to stop it.
In the absence of selection pressure the override genes will mutate randomly. The end state of that process are broken genes that don't do much of anything (loss of function), and so the color goes back to the default.
For human eyes, the default is blue eyes and there are then some genes that overrides this by producing melanin, which turns the eyes brown.
After all, one of the things that's so hard for us to grok about evolution is that it's not a goal-oriented process in any way. It's not engineering, it's the process of random chance and what can persist.
For example, a lot of people think that all our junk DNA actually does something profound and important, and while some of it probably does have a purpose we don't yet understand, I think it's almost certain that any process that makes a bunch of random decisions that are mindlessly culled by environmental forces will inevitably generate some flotsam and noise in its code. It's kind of like training AI; it definitely won't give you the most minimalist method of carrying out that task, but it will eventually find some way to carry it out. Almost as a brutal, automatic tropism.