Edit for clarity: when you do this, your finger is not actually "on" the slider handle itself. The slider shows up in its usual place, which is offset from where your finger is. But the sliding still adjusts it.
It's a shame that this is not intuitive or discoverable. Its much more efficient but I had to have someone tell me about it.
I guess there's a still tradeoff: this way you need to move your finger around, but you can immediately skip to exactly the level you want. With holding a button down you have to time it to get to the right spot instead.
Personally I like the slider better but that's just my opinion.
If that bugs you enough to bother with a workaround, you could try something like this using HammerSpoon: https://gist.github.com/spinscale/fd82f00da29447990f27f36b3f...