A lot of it has to do with the electrical properties of such a switch. Even older versions of DisplayPort have massive data rates, meaning each electrical signal switches many millions of times a second. Also, each signal may have to switch states close to the same time as the other signals, meaning there needs to be some sort of matching between the different signal paths.

Add in noise sources from the power supplies, your cell phone, and other sources, and it becomes a challenging physics problem. Not to mention, you don't want those signals leaking out either. At these data rates, wires are both transmitting and receiving antennas.

Switches are inherently very mechanical devices, so designing one that can operate in these conditions reliably would be both complex and expensive. Much easier to have some silicon do it for you.