This extension doesn't use selenium, plus that's not entirely true, selenium sees HTML and DOM while a user sees the final render; there's ways you can hide content from a user while showing it to Selenium-style bots.

I thought selenium will throw an exception when an element being clicked is not actually visible.

If “ignore all clicks from a user that clicks >3 ads on a page” isn’t good enough for an ad network, it can add three or four ‘ads’ that technically are visible, but the same color as the page background. If a user clicks a few of them, ignore all clicks from that user on that page.

AdNauseam could detect that, too, but it gets exceedingly hard, slowing down the user’s browser. So, I think the ad network can win that battle.

Does it? I don't think you can reliably identify whether something is visible if the other site, which controls the CSS, does not want you to. It's a classical arms race situation.

Selenium can reliably do that, as it is a proper webbrowser controlled through an API

Yes, but it is not a human eye and brain.