The right way to think about these kind of things is in terms of responsiveness. For example, when I think of any request intiated by a user, there is a ~200ms budget of time to service that request. If you can shave off a couple hundred micro seconds, those can be spent on returning a better answer.

Everyone has a piece to contribute to servicing user requests, and every should be mindful how much of the budget they consume. Using select (or epoll) more efficiently makes everyone elses job easier. It is important for everyone in the pipeline to think this way, you included. The user experience depends on it.

In a nutshell: performance still matters because your program isn't the only one running on the computer.