It's enforced through visibility / rankings in Google Search results.

https://developers.google.com/search/docs/guides/about-amp

Isn't there a term for using dominance in one market to compel behavior in another?

Your link doesn't say that Google forces AMP on publishers. It shows that Google displays an icon next to AMP results for mobile searches to indicate the page is mobile friendly. Bing does the exact same thing: https://blogs.bing.com/Webmaster-Blog/September-2018/Introdu...

This is not enforcing AMP on publishers in the results, and the argument that it is by using icons falls under the 'it's kind of the same' category.

It says they use site speed in ranking, and, well, I'm sure it would come as no surprise if Google's (largely) having served a page/site increases the speed just enough.

If you can beat AMP load times, you will likely rank higher than AMP pages from the speed factor.

This is how you do agnostic search results, and it is not putting the thumb on the scale as you're implying.

There is a term for that.... I think it's either it's "abuse of dominance" or "anti-competitive behavior"

to put it as plainly as possible, it's absolutely hilarious that tech people buy into these insane myths about AMP, there's a reason why no serious antitrust person brings it up, it's fighting on Google's territory - it's a wide open standard, used throughout the industry, formed in response to proprietary solutions designed to tax suppliers by Facebook and Apple, immediately and fully shared with competitors.