Most of the downsides would be on the goods side. Swiss companies would loose market access and the chance of "better" trade agreements is even worse then the UK, especially currently.