Before each call to select, you'll have to FD_ZERO the fd set(s), then individually FD_SET each fd you're interested in.

If that's a large number of fds, then that's quite some overhead.