I mean if a business if built on top of your business, is it actually illegal to say "Pay me if you want continued access?" Is it the wording that makes it illegal?

Who built what business on whose business? It's not clear what you're referring to.

... to the open source project WordPress? 1) it'd be hard to forbid any one entity from using it, based on the licensing, and 2) the announcements and banners pushed to users referenced WP.com, Matt's for-profit competitor which WPengine doesn't use, and brings about elements of tortious interference.

Extortion does not look like an easy case to make. Pull up some of the state statutes: they all seem to have intent and malice requirements, and/or, like California, require the threat to be of an unlawful injury. Threatening to ruthlessly exploit capabilities you lawfully have, like the bully pulpit of leading the WordPress project or the strictness with which you license your trademark, is unlikely to meet that standard.

(But see below for the 'DannyBee comment on how UCL unfair competition might work even if you can't make a case under the extortion statute itself).