The open source remote extensions doesn't support hardware keys as ssh keys. They have an issue open for this for years now.

I use vscodium either way, you need to get used to stuff like this from time to time.