In the mid-90s, years after IRC was written, I set up a distributed system for the financial traders I was working for. Our between-cities links were 64 kb/s guranteed and could burst all the way up to 256 kb/s. And those links were not super reliable. These were connecting systems with Pentium processor running ~90 MHz with ~8 MB RAM. They did very, very little compared with even the cheapest server slice you can get with AWS.
This is one of those things, like George Washington never knowing about dinosaurs, where it's just hard to comprehend how people thought back in the olden days.
> [...] clients could be configured to quickly attach to another server in another region and that is just DNS management. This could have been further improved by amending or replacing the IRC RFC's to allow SRV records. This may have been done by now for all I know.
To set the stage: Larger IRC networks balance their global servers. A DNS A query for irc.example.com will yield a list of geographically local servers, possibly shuffled on each query as well.
I know of at least one IRC network that refuses to send even the list of all geographically local servers, only sending a subset, as a measure to avoid trivial DDoS attacks if people don't go around collecting the DNS records ahead of time. I'm told that this actually works because the thread actors are not the sophisticated kind.
Incidentally, I have also noted that some networks will shuffle the order of A records for each query because the clients cannot be trusted to select a random DNS response. Considering something this trivial already doesn't work, I dread to imagine how much a DNS SRV implementation would go wrong, considering it needs both sorting and a weighted random sampling[1] to really work.
[1] https://datatracker.ietf.org/doc/html/rfc2782 page 3 et seq.
OTOH, current IRC usage numbers are pretty low, a beefy single server should work, except for the disruption potential of single servers. Latency can be a bit of an issue too, depending on where your users are; not great if users are in south asia and the only server is in east coast US.
an ircd with a few thousand clients was cpu bound on poll()/select()
/dev/poll and kqueue/epoll were game changing
But few knew the trick so it didn't catch on.
with poll()/select() I don't see how you can avoid checking every FD at least once (poll's fd counter aside), vs. epoll() only returning those in the desired state
(and I don't think you could do tricks like epoll_wait() on an epoll fd)
Fork some child processes, and keep an AF_UNIX socketpair() open to them so you can pass them file descriptiors with SCM_RIGHTS.
Have the main process divide up the fds it is waiting on into a "hot" subset and cold subsets of size at most N, and for each cold subset pick a child process P. fds can be moved between hot and cold at any time, and generally you will move them to hot after they have woken and been used, and move them to cold after a few consecutive poll-cycles where they were not ready. Don't move fds to cold subsets belonging to child processes that you don't want to wake, though.
When the main process is ready to "poll everything", have it iterate over each child process that is not already sleeping, and send a message over the socketpair(), containing a list of fd_set additions and removals to that child's wait-for subset, including the type of poll (read, write, etc).
For each fd where the child doesn't have the real file descriptor yet, pass that over the socketpair() as part of the message. (If threads are usable instead of processes, there's no need to send the file descriptor. But on old systems, the system threads were often implemented by userspace multiplexing with poll/select anyway, so it wasn't a good idea to use threads with this technique.)
As well as a list of changes, this message tells the child process to run poll/select on its subset, and then reply with the set of fds that are ready (and their readiness type).
After issuing all the child process messages, the main process does its own poll/select, to wait for hot fds and replies from the child processes.
The reason this has different scaling properties, despite the overheads, is that each child handles a limited size subset, messages scale with the amount of change activity not the size of sets, and ideally the "coldest" fds end up gathered together in child processes that continue to sleep between a large number of main process polls, so the number of active child processes and messages scales with the amount of change activity as well.
Keep in mind, even active fds are removed from the wait-for subset if they've recently reported they are ready and the poll loop hasn't read/written them yet. So it has similar algorithmic properties to epoll.
As a bonus in the case of select(), the fds in the child processes have smaller values than in the main process. So in addition to the number of fds polled per cycle scaling with the amount of activity instead of the total number of fds, the fd_set bitset size does not grow with the total number of fds either. In the main process the bitset size does grow, but it's possible to juggle fd values with dup2() to overcome that.
that is indeed clever and I agree would practically improve the performance drastically for many (most?) workloads
worst case scenario (say all your sockets have a approximately the same chance of waking up): the behaviour devolves into the same as for poll() though?
as a broadcast medium: IRC has some annoying qualities in this regard (inline PINGs at regular intervals, events visible by nearly all that trigger socket avalanches, etc)
To picture it, maybe it helps to consider low and high uniform probabilities. If the uniform probability is a low enough per wait call, epoll/kqueue/etc have low cost per wait call, but so does this child-select method, because most child processes don't wake on each wait call. If the uniform probability is high enough, epoll/kqueue/etc return most of the fds, scaling just like the cost of select/poll, and child-select. So epoll/kqueue/etc and child-select match at the extremes, modulo constant factors.
In general, with a constant-bounded subset size, you can say the cost of unnecessary fd polls is amortised over the necessary fd polls, because unnecessary fd polls only occur in child processes that wake due to a necessary fd poll. So the algorithmic big-O scaling of this method is still the same as epoll/kqueue/etc as the number of fds increases and the probability varies, whether it's uniform or non-uniform.
No guarantees about the constant factors, though. You'd still want to tune parameters, to have enough and not too many in each subset. Constant-bounded subset size ensures scaling but not necessarily optimal performance. You can relax the constant bound and still keep the worst-case big-O scaling, if it's adaptive in the right sort of way, to better handle non-uniform probabilities with fewer processes.
Every network also seemed to have its own fork of ircd (and at least one rewrote it from scratch), so could have just been that the ones I used patched it in and others didn’t.
A slight correction: Services normally link as a server to the network, which is how they get the higher privilege that they do (because only servers, not clients, get the ability to kill users from the network, etc).
And to add to this for others who may be curious: typically there is some special configuration on the IRC server side to allow the link, and some additional configuration to disallow clients from changing their nickname to names like "NickServ", etc (but to still allow the names when a server on the network broadcasts a user with that nick). Normal non-Services IRC bots, on the other hand, connect as regular clients.
[1] https://en.wikipedia.org/wiki/IRC_takeover#Riding_the_split
Part of the answer is historical and part of this was technical. IRC has been around for a very long time. As such, the earlier versions of the servers and daemons could not accept tens of thousands of client connections ePoll vs Select. The connections between servers are multiplexed and not directly related to the number of people connected to the server. There was also a matter of latency. Servers in a region would keep the messages local to that region, as only people in the same channel get the messages and it was less common to have people in the same channel all over the world. This also changed with time. If there was a split, you lost other regions. This was not always the case, so of course I am over-generalizing since there were many different IRC networks designed by many different people. Being long running services, I had seen a great deal of hesitation to re-architect anything on the fly on at least some of the networks, even after ePoll and modern hardware made it possible to have tens of thousands of people on one server. Some of the smaller IRC networks indeed consolidated into fewer or a single server.
Another facet is logistics and ownership. Many of the bigger networks are comprised of servers owned and managed by different people and organizations. The servers are linked as a matter of trust. That trust can be revoked. Most of the early IRC networks were run by people doing this in their free time with their own money and/or limited resources. In some other cases some organizations prefer to have their own servers so that their own people indeed to not suffer splits for their local communication. There are a myriad of other use-cases and reasons why some organizations had their own servers. Sometimes there was a need to give LocalOps special permissions that would not be permitted network-wide. Despite the technical capability to have less servers, some organizations are not going to give up their local nodes.
One issue not mentioned is permission losses on splits. The issue with splits and permission changes has more to do with the way services are integrated into IRC, or more specifically, aren't. Services are treated like bots with higher privilege and most if not all of them were not written to be multi-master. Rather than dealing with moving services around or pushing for read-only daemons, they just lived with the possibility that there would be splits and they would eventually resolve themselves. I personally would have preferred to see a more common integration with OpenLDAP. Some of the IRC daemons can use LDAP, but it is more of an after-thought, or bolt on. This would have allowed splits to occur without losing channel permissions and clients could be configured to quickly attach to another server in another region and that is just DNS management. This could have been further improved by amending or replacing the IRC RFC's to allow SRV records. This may have been done by now for all I know. I shut down my last public server some time ago.
There is a lot more to this than I could sum up on HN. Anyway, today you can fire up an IRCd of your choice on modern hardware and accept tens of thousands if not hundreds of thousands of people on a single server if you wish. It is technically possible. I would still design the network to have multiple servers, as you will eventually hit a bottleneck. If you really want to do this, you will have to de-tune the anti-ddos counter measures to allow the thundering herd to join your standby server or make code changes to permit the thundering herd briefly on fail-over.