Really bad.
Invalid curve attacks will leak your secret key, which most old implementations (i.e. the ones likely to end up in your hardware) don't protect against.
If you reuse an ECDSA nonce, you will leak your secret key.
If you have even the smallest timing leak, guess what.
https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2016-7056
https://web-in-security.blogspot.com/2015/09/practical-inval...
Essentially, a bunch of ECC implementations didn't do proper bounds checking and leaked data. ECC still offers a lot of benefits in terms of key size, performance, and security, however, so do your research carefully.
If you're doing ECDSA or ECDH over P256, it might be fine. It might also be a complete disaster. So, we don't recommend it -- and if you must, you should have someone audit it for you, because you're not in the safe zone anymore.
(Examples of disastrous failures: P256 off-curve attacks, unsafe k choices for ECDSA.)
However, this article says "Don’t do ECDH with the NIST curves" and "really, especially avoid conventional DSA and ECDSA".
How bad is it really? I mean, the article's pretty unequivocal, it sounds like using that crypto element isn't a very good idea?
[1] Motivation: Users can completely reprogram application processor without wiping (or being able to extract) the private key. [2] https://www.microchip.com/wwwproducts/en/ATECC508A