I've actually had this too, but the other use-case is more common for me (I adjust volume using the buttons, and rarely change brightness). I think it's the flat-UI thing where it's hard to see the boundaries of the slider element - so I would maybe not swipe near it if there was a little boundary, and you wouldn't swipe outside it.