The least you could do is acknowledge to the person behind you that you're about to do it. You're acting as if you're being told to file a permit. On the contrary, it's pretty common courtesy.