Proof: You will have 0 memory free and start using swap when the restart happens - type 'top' and watch this go down.
Fix: Use less threads. The number of concurrent users you can support is the # of pages you can serve up per second in each thread times the number of threads your server can handle. So a huge improvement can be caching. Cache pages, blocks, hell maybe your whole site. The cache hits will serve 10-100 times more responses per second than generating your entire page.
I consulted on a project that was having horrible server performance on a huge server with relatively static data. The way they produced pages was really poor and took 3 or 4 seconds. There is no way you can serve up a large number of concurrent requests with the turnaround time over a second. I added caching as a quick fix and they ended up just being happy that 99.9% of their pages served up in 1/100th of a second.
The blurb from that site:
Munin is a networked resource monitoring tool that can help analyze resource trends and "what just happened to kill our performance?" problems. It is designed to be very plug and play. A default installation provides a lot of graphs with almost no work.
One day, your software had a resource leak and was eating up too much of memory/file handles/etc... One of your developers tried restarting the system and like magic, it fixed it. After a few days of having to do this, he got tired and simply put the restart command into an hourly cron job or a script somewhere on the system.
I've seen it a thousand times. My consulting rate is $200 an hour. If this turns out to be the problem, I'll expect $6.67 in the mail from you. :-)