You're right that in and of themselves, the cosmology of Genesis don't amount to much. But there's a real concern that if parts of the Bible are not trustworthy, other parts may also be untrustworthy. And a Bible you can pick your favorite parts from is practically dangerous. This is how things like chattel slavery were justified by "Christians".

There are also theological implications to editing your own holy book. You end up editing the book down to the God you want (who, strangely, thinks like you and has the same friends and enemies) instead of the God who exists. Of course, you can make those objections to the various canonical versions of the Bible. That's an interesting digression with good answers, but a digression nonetheless.

The point is lifestyle change. I'm saying ignoring this part of the Bible may not affect much there, but ignoring certain parts of the Bible certainly does. And how do we resolve the "who watches the watchers" problem? I guess you could say "only cosmological stuff doesn't matter too much" and be pretty much in line with a lot of Christians.