To me this indicates some pages loads are blocking. I've seen this before where page loads triggered blocking network calls (web services) and the web service client code wasn't configured to use timeouts.
I'd take a look for that.
Another thing you can do is to treat the symptoms. Setup a cron job that runs every X minutes and kills the Y oldest threads.
From memory there is a config setting in Apache that specifies the number of times a thread should be reused, so try tweaking that, too.
(Edit: if this is the case, then switching to Ngnix won't solve it. You really need to understand why that threadcount is rising)
Threads or processes? If using the worker mpm, switch to prefork.
If problems persist, strace a frozen process to see what it's doing. strace is an incredibly useful diagnostic tool.
You could even set up a script to parse the server-status page, find out which processes are hung based on the SS column (seconds since beginning of most recent request) and kill that PID... but that's not pretty.
The site is hosted at Slicehost, on two slices.
Webserver: 4GB slice. DB server: 2GB slice.
We do about 3-5k uniques/day.
I've tried scaling the DB server up to 4GB to see if that solved the problem, but we continued crashing every hour.
Part of the problem though with my own efforts is that I don't have much experience with sysadmin stuff to be able to say for sure that any of my own observations and attempts to solve were done properly. For instance, I scaled up the slice, but maybe I didn't restart Apache in the right way afterward to know if that made a difference or not.
Can you describe as best as you know, your hosting environment?
1. After a restart of apache: Apache starts out running. He spins up a few threads, 10 or so and then kills a few as he goes on his merry way.
2. Eventually, the number of threads starts rising. 20 threads. 40 threads. 60 threads. 90 threads. 130 threads.
3. 45-55 minutes after the previous restart, apache is now frozen. SupervisorD, which has been checking every 5 seconds to make sure apache is ok, spots this, and restarts apache. The site shows a fail-whale image for 5-10 seconds as SupervisorD does its thing. Then apache comes back online and we repeat the cycle.
4. About every 3 days, the semaphores pile up, and SupervisorD is unable to restart apache. Apache goes into FATAL mode and I have to run a funky command line command a friend showed me to clear out the semaphores so Apache can be restarted and taken out of its tailspin.