Your breaks or has minimal functionality if the CDN is down.
Your site loads slowly if the CDN is under load
Your users are now leaked to the CDN (privacy issue)
Of course, if the resource is cached, it's less of an issue, but it's still an issue. If you don't care about those things, go ahead.
How about 0kb?
Whatever happened to shared-resource URLs, where there was a good chance the library was already cached?