Cell phones operate in licensed radio spectrum, so they need to have proper testing and certification (https://www.fcc.gov/oet/ea/rfdevice). Any device not properly certified would be illegal to manufacture or import into the US.
Separately cellular networks require PTCRB certification of the devices to ensure they are interoperable with the network (https://www.ptcrb.com/). The FSF could in theory write custom firmware for baseband and wifi chips, but they would need to seek certification as this would be considered a substantial modification. It would likely require cooperation from the chip manufactures to provide samples with various testing/debugging harnesses enabled.
Qualcomm and the like would probably sue to stop the FSF on the basis that it could put their own device certifications into jeopardy.
That isn't even touching on non-transmitting components like GPUs or sensors where the actual functional logic may be split between hardware and software (your blob driver). Even by doing a clean room reimplementation, you risk infringing on software patents, and will have little flexibility to work around them since the hardware will expect things to be done a specific way.
You would think it would be ridiculous to assume the people working on this know nothing about the mobile space, yet their actions do bring that into question.
All that patent and legal business is probably a more important/existential concern and a go/nogo-factor if you want to be a commercial player in a market-driven environment and less so for an entity like the FSF.
Second, fuck the carriers. Certifications will not persist as soon as real Foss phones are available. Nothing persists against a world of free hardware invading a realm. And even if: freeing everything around a modem blob would still be a big step forward.
It's frankly ridiculous to assume the people working on this and the organisation that already supported replicant knowns nothing about the mobile space.