Not necessarily. It is entirely possible to force-port a number by faking a letter of authorization as OP described. When I worked in telco, I did LOA-based paper ports at least twice a month for customers whose losing providers were being jerks* about not coughing up approval. The number port database admin at the time only required that I tick a box on an electronic form certifying that I had written authorization from the customer to force the port and upload a PDF. The losing carrier could do nothing to stop the port.

Your Google account security actually means relatively little here for a determined attacker. Sure, it prevents the takes-only-a-few-minutes automatically-approved ports (which, yes, is what roughly 90% of attackers are going to go for) so you're excluded as low-hanging fruit but Google is not itself a telco and doesn't host the phone numbers directly on its infrastructure so it can't do much to harden against this kind of attack.

* (A certain wireless provider that used to be a three-letter acronym and is now fairly low in the alphabet was notorious for refusing ports when wireless number portability first started.)