(It's still a right by the way, as I said it can be both)
Edit: Anyway, I don't feel the need to argue this since it's just a semantic argument but the general thesis is that I found your narrow definition of privilege to be surprising compared to how I've experienced people using it but English differs over geography so idk.
When you have a right to free speech and then someone puts you in jail for exercising it, it isn't that you don't have the right, it's that your right is being infringed.
Calling it a privilege that you don't have any time someone is infringing on your rights would mean that nobody's rights could ever be infringed, since any time they were infringed it would merely mean that you didn't have them to begin with. But you do. People have human rights even when someone is violating them.
Due process is a human right in a way that having a title is not.