Most of the insecurity stems from the baseband. It's a black box processor with unfettered control. Once you get rid of that, the rest of the hardware is more like a general computer. Getting PCs to work with an entirely free stack is also hard, but tractable. Replicant already does a good job with some old phones.