> I know people fluent in sign (a large part of their job is working with the deaf), and when they sign to babies the babies are like "you idiot: I just heard you talking to someone else so don't insult my intelligence by not talking to me"

The way my friends have done this is to speak and sign, not just sign. I don't think any of them would've thought it was appropriate to only sign to a hearing child. The purpose was to teach them both the verbal language and a way to communicate back while they were still too young to verbalize (or, in one critical case, unable to speak due to a birth defect but were perfectly capable of hearing).