I never read it as "removing oversight", more like "removing excessive admin". And it explicitly says to value "individuals and interactions". Part of that means you need a facilitator in retrospectives who can detect conflicts between individuals which may not be being addressed in that forum - and ensure that they are addressed elsewhere.