Your analogy is spot on. It's a perverse incentive system. It doesn't make any rational sense, but that's the way it is. Even more ironically, the impact factor of open access journals matters even more than for paywall journals. Once a paywall journal is established it's fairly stable. If it loses its impact factor that's bad, but doesn't mean the death of the journal. But with an OA journal if you have an impact factor and then lose it, your submissions drop off a cliff, which directly ties into your revenue, and that will often be the actual death of the journal.