Does the master process actually accept() connections while the workers are being restarted? If not, clients will eventually fail to connect after the accept queue length has been reached. This can happen very quickly as accept queues tend to be pretty short (in the hundreds, if not less).