This is not using it as a second factor. It is using it as the only factor. Having SMS as the only factor is not purely additive. As such it can (and obviously does) reduce security.
Account recovery is hard, SMS is quite usable there, but way to insecure to be the only basis for bootstrapping account recovery.
Let's say that you remember your password, but your house just burned down. You cannot replace the U2F keys and backup codes that were lost in flames. But you almost certainly can bootstrap your real life identity far enough to get a replacement SIM.
Which, in combination with your password, should be enough to get your digital identity back.
But indeed, sms as a second factor is much easier to recover in catastrophic situations than some other second factors. That is a fair point, and an advantage of sms over other common second factors.
You are forgetting social engineering. Humans find it reassuring that the security process happened as usual, even if in fact the apparently "usual" process was them being being phished. This can mean they're actually less alert than they would be otherwise.
You get an urgent message from your bank about an unexpected $500 transaction, you follow the link & you need to enter your password as usual of course, and then it tells you that you'll get an SMS and to type in the code so you do so. Phew! Disaster averted! Right? This must have been real, you even got an SMS from the bank.
Alas the SMS was from your bank, and the bad guys didn't have a way to intercept it, but they didn't need one because you typed it into their phishing website. That unexpected $500 transaction wasn't real, but their emptying of your bank account will be.
"You get an urgent message from your bank about an unexpected $500 transaction, you follow the link & you need to enter your password as usual of course. It was a phishing website. Your bank account will be emptied."
It did not reduce security.
It most certainly can reduce security, that's the point. If I don't have a phone number on my account (which I almost universally don't) then no amount of SMS hijacking will ever matter.
If some provider forces me to put a phone number in, now I may be vulnerable to a weakness I didn't want to be vulnerable to. Maaybe today that particular provider uses SMS in a stricly additive sense. Maybe. Just as likely next month they'll redesign their site to be "easier" and add back the vulnerability.
Same with recovery questions. They make the security stricly worse for most people since they are password-equivalents with far lower entropy. Although personally my best friend from high school was named D3ho9WvylJkws1zfAKUxZjdYuCsS.
I responded to this in another post.
> There is pretty much no form of second factor that users are worse at passing than backup codes.
Agreed, I also mentioned backup U2F. At this point modern smart phones package TPMs that can also do attestation, so we're really not too far away from being in a situation where the vast majority of people have a U2F token in their pocket.
There is pretty much no form of second factor that users are worse at passing than backup codes. Even if people print them out (few do), they won't find them when the emergency happens. You need some form of trust that can be bootstrapped again from scratch.
For most of the world, SMS is it. The Nordic countries have the bank if system. But the market is too small. Hopefully the EU-wide identity verification systems solve the scale problem.