What you describe is the "common sense" explanation of the origin of money that we hear a lot, "even peasants need to trade with each other, and maybe they don't always have what each other needs directly, they need an abstract unit of value." But it's unclear if it's actually based in any historical evidence or was just invented as something that "made sense". Graeber suggested it is not based in much historical evidence, and that there is evidence for other explanations.

I think Graeber's theory of the origin of money may not be universally accepted (not sure how accepted it is); but I think it's true that the 'common sense' theory isn't actually based on much historical evidence, but is just "an argument that can be made", as they say, which somehow became the "common sense". That's not a very reliable way to accurately describe history. All sorts of things that people in different times assumed "must be true because it just makes the most sense" turn out not to be. "seems simplistic" or "seems more plausible to me" is not a reliable guide for distinguishing truth from fiction, and what seems more plausible to me may not to you.