WordPress itself, as in the source code, can definitely handle however many visitors gets thrown at it. Where you'd have to scale is the amount of RAM where your website sits. I've been pretty successful with testing 10,000 visitors over a minute period on 512 MB of RAM without issues, though I recommend you have at least Memcached installed and enabled.

If you're talking about 100,000+ visitors, than you'd probably want to get some good advertising and update your RAM.