I think there would be a different answer depending on who you ask, but some of the biggest downsides as of now:

1. Lack of DANE/TLSA support in browsers

2. Lack of SSL since the CAs aren't issuing for Handshake Names. This is independently solved by #1.

Solving #1 is interesting because it would also be useful if you didn't care to use a Handshake name and also didn't want to depend on the traditional CA system. It could save the need to for companies to ever install a certificate on their employee's machines along with coordination around updating the certificate.