You're right, the "it's just a number" argument doesn't hold up. What's interesting in this case though, is that it's not just any other number, but a prime of mathematical relevance.