The advantage of U2F is that it isn’t phishable. You can only sign the message for the pre enrolled URL.
Yes, you can still fall for more elaborate instructions but you cannot simply give the attacker your credentials through a normal looking flow.
I also disagree it's that far fetched to get people who'd do that to also do whatever else you want them to.
And while SMS swapping is miniscule in comparison, the big difference there is that there is no signal at all that you are under attack. With phishing, there is no way you are not feeling something is at least a bit off, so you know to check soon after, even if you've been compromised.
After this happens it takes you to a “something went wrong” page and has a link back to your real bank website.
With U2F this impossible because you cannot sign a message for bank.com when visiting evil.com.
Any access is easily "phished" with pliable people (which is not necessarily a set of people, but also a question of timing and circumstances: everyone is sometimes more or less pliable): "please log in with your U2F device, download that document and upload it to this URL https://your-company-confidential.s3.amazonaws.myurl.com/, before we can reinstate your access to company systems".