If your manager signed you up for pager duty and it's not considerd one of your job requirements, you should unsign yourself up and also have a chat with said manager (never sign somebody up for pager duty without first having The Talk with them).

You have to push back hard now, or they will continue to take advantage of you. My managers at Google tried to do this and they always ignored the on-call pay bump, so it was easy for me to say "this isn't on-call" and not really do it.