I think the key there is this section: "Central to the construct of emotional neglect is a failure to meet the basic emotional needs of the child. Key indices of early emotional neglect include both a pervasive parental failure to respond to children’s signals, particularly those of stress or distress, and a pervasive failure of parental initiative in proactively structuring the interaction with the child in protective and developmentally enhancing ways."

I honestly think whatever parent(s) are available just need to show their children that they are loved & respected. From the sound of it, I would agree with your first instinct - it should balance out. But, if you can, having loving grandparents around is supposed to be good for little ones too! :)