It's not a matter of "stopping reading". Impact factor metrics are based on number of citations. And if I publish a paper on a new method to do X, I am expected to compare to (and thus cite) previous papers that did X. I cannot just decide not to cite relevant previous work because it is published in an Elsevier journal.

Of course, sending papers to OA journals, going the extra mile to cite them, and preferring them to non-OA alternatives when any paper will do (e.g. when you are just providing some examples of applications of X, without being exhaustive, and you can find some in OA and others in non-OA journals) does help.