You point to fundamentally different assumptions, and as a whole I agree with that too. I think that anarchists put less emphasis on human nature per se than they do on ethical organization practices--they preach a scheme of purely voluntary self-organization mostly from an ethical standpoint, and this stance implies that they view human agency as the most important ethical question, rather than meeting basic needs (communism) or castrating exploitative practices in labor (socialism/Marxism). Libertarians seem to be whistling another tune, more along the lines of anarchism but by also tying capitalist success to a person's ability to steer their path in life. Anarchists think that free association will provide for those who need by virtue of a sense of humanistic camaraderie.
I'm not arguing for or against anything here or anything you've said, and I'm not sure I'm doing much at all except a fun exercise to clarify the varying motivations of people in their quest to restructure economies and societies. But it is interesting to consider and feels good to put to words.
I think, from your statement, that you agree with the above, and you say this is evidence that the right-left axis is not useful. I personally think it might have to do with anarchism making fundamentally different assumptions about humanity and human nature, assumptions that are so out of phase with my understanding of humanity that I... just don't see it in the ways that I can understand a marxist or a libertarian. It's not about agreeing or disagreeing, it is like we have had completely different experiences of humanity and completely different readings of history.