Love this, I totally agree. That's why we want to find a way to make this work without algorithms. We know what hits us in the same way as something else. I'd love to build this site to achieve close to what you suggest.
You're wanting https://hangout.fm/
It's kind of a meta music discovery, I can piggy back off others who have tastes that I like.
The biggest benefit of this is that I don't necessarily get recommendations of music that's adjacent to what I already like. In other words, if I like Depeche Mode, I don't want to listen to bands that sound vaguely like them.
What I've yet to see someone do, is track what music someone listens to after listening to a song. Often when I hear a piece of music, it makes me want to listen to something else that feels the same way or reminds me of being in similar place in time. A music recommendation engine based of this concept would be interesting.