I have my doubts about select causing 100% cpu utilization, I suspect you were doing other suboptimal things as well. The sample code I wrote is running well with both 100 and 10,000 connections. I have my own anecdotal evidence where application was barely handling just 100 mostly inactive connections and OPS guys suggested limits of just 50 connections per application. After I fixed how feed sets were created and how the result of select was processed the same application was running just fine with 8,000 connections. We had to support Linux, Solaris, AIX and HP-UX at that time and select/poll were available on all of them. That's why I invested time in optimizing the code instead of switching to epoll. OPS guys still suggested limit of 1,000 connections per application but this time it was for availability and other non-performance reasons.
For most people, if your program is handling 10s or 100s of concurrent connections, select() should be just fine.
If you are going to be handling more, it's worthwhile looking into the other syscalls to improve performance. In any case, I'd recommend abstracting away the event handling so that you can switch between different syscalls and can use benchmarking to try to replicate the traffic you want to handle. There's lots of libraries that will do this work for you, unless you need to get into the low-level stuff.
Trying to judge the syscall performance based on reasoning about amount of data transferred between your program and the OS, number of syscalls etc, is very difficult. You are much better off measuring the actual behaviour. For example, epoll() seems like a terribly designed API to me as it involves making many syscalls (whereas kqueue() is just one per loop). However, I found epoll() was very high performance. I guess the cost of syscalls on Linux can be very low in some cases.
The kernels real range and userland's virtual range won't overlap so for a lot of functions kernel memory just has to mapped/unmapped on call, not invalidate _all_ userland bindings.
Well okay they will overlap. So yeah your mappings may get invalidated but for synchronized higher performance systems calls they _shouldnt_.
This lets them be in the 10's to 100's of nano-seconds.
Normally the most _expensive_ part of a linux syscall is the TLB misses after one.
---
Your model of memory transfer size assumes data is being copied.
The Linux kernel has a lot of features to let userland, devices, and itself all share the same memory copy free.
Kernel address space and userspace address space don't overlap at all. There's zero mucking with memory mappings, TLB etc. on syscall.
If you're using nonblocking I/O, or threads, select can become the bottleneck with a few thousand interesting connections at sizes that are much less than the FD_SETSIZE max.
The user-space CPU utilization should be near-0 since this is really a kernel I/O problem; the fact that it is much higher is an indication that a lot of wasted user space work is being done.
Select, by design, has a flaw: the kernel must scan the bitmask for the highest-numbered fd passed in, even if there is little change in the fd_set bitmask (as is usually the case). This means incurring a cost if any fd in the set has a high fd number. The fd on which you accept probably has a low number, but your client I/O fds will have increasing numbers as the server gets greater and greater utilization.
kqueue (and epoll, but I haven't used it) resolve this problem nicely. Only things that change state must be communicated. poll is better than select in this respect if the array is sparse, but if a large number of fds are interesting, it will also get expensive.