You can use a dynamically sized thread pool. E.g. remove a thread with a certain probability once it's idle for more than X seconds.