Not only GitHub but the specific user's account could be compromised through e.g. a weak or reused password.

In the general case, you should be fine. Even more so if the person whose GitHub account you are authorizing regularly checks their keys for suspicious activity.