The basic building blocks can actually work great, it's often how they are executed and how many places go very much micromanagement wrapped in agile's skin while also instituting lots and lots of meetings - total opposite of what both Agile and XP advocated.
Some of the best agile/scrum I ever worked under came from "Scrum master" who explicitly stated that one of his jobs was to see what approaches work and change what doesn't work for the team.
unfortunately from my experience, "what works" can vary wildly from person to person. One person wants to be given a task and left alone to understand it and write some throwaway code as part of the understanding process. another person wants to bikeshed it to death first before letting you touch any code. so the first person has to hide the work on the prototype and pretend to come up with the insights for the first time during the bikeshedding session.
Sometimes you can compromise, sometimes you can figure a proper allocation of resources, sometimes you can part amicably instead of murderously.
Ironically, if I were to give one-sentence summary of what the idea behind Agile is, it would be exactly that: "see what approaches work and change what doesn't work for the team". Running through that feedback loop rapidly is the whole point of Agile, and the one thing that actually distinguishes it from typical management methods at the time (as opposed to silly comparisons with "waterfall" boogeyman that never even existed).
Everything else is process - the thing that's supposed to be sculpted. Focusing on that is committing the sin explicitly called out in the Manifesto - putting process over people.
If you're phrasing it like that I've been doing agile since way before anyone invented the term.
But that's only 10% of cult agile, which is what is practiced generally.
Picking what to work on for [time] without external additions is a pretty good improvement over getting new assignments daily.
Or maybe you're just doing it wrong, I'm not here to defend anything codified in Agile, I'm just saying that elements of what most people call scrum has been the best I've seen in over 20 years. Maybe I just missed the days of locking yourself in your single office.