Non-market interactions are invisible to conventional economics because they're not priced, but they're by no means insignificant. They range from developing-world peasants farming land to which they simply have no title deeds, through to the form of communist praxis which is the dominant social structure of the supposedly free-market west -- the nuclear family. (Or do you present your kids with a bill for their personal care and feeding? Non-dysfunctional families run along lines Marx described as, "to each according to their needs, from each according to their abilities": sharing without reference to individual wealth is explicit in state-recognized marriage vows. That the power/money relationship often reverses a generation later, as adult children support or nurse their aged parents through their last year, just underlines the pattern.)
Graeber, as an anthropologist, is interested in all human interactions, not just marketized ones. And his particular field has been the intersection of the market sector with the non-marketized greater cloud of human relationships.
I thought it was a stretch for Graeber to basically call everything socialism in the (roughly speaking) second section of Debt - especially since there is a categorical distinction between Marx's tagline ("to each... from each...") to what actually happens (in families etc), which is, "to you according to what I perceive to be your need, from me, according to what I can afford to give".