We are much of the same mind. The organ may be the queen of the instruments, but the human voice is king, and I much prefer singing without distracting organ accompaniments.

That being said, I don't think you can get a full picture of Bach's achievements without having some appreciation for his organ work, particularly the chorale preludes. Those are some of the most magnificent pieces he ever wrote, and I adore them despite my ambivalence towards the sound of the organs.