The performance hit is because Thunderbolt has at most PCIe x4 bandwidth, while internal GPUs are connected to PCIe x16 lanes. That's a massive loss of bandwidth.

But it also tells you a lot about why Framework didn't make x16 bus hot swappable, it's a significantly bigger engineering challenge.