News for hackers. Stuff that matters.
Re:Why is IRC distributed across multiple servers? by H8crilA on Sunday September 12, 2021 @01:31PM (#28503062)
Just remember that "netsplits" exist in every distributed system, be it a chat app or a database. It's just the CAP theorem. IRC has chosen to sacrifice C (consistency).

The only thing that changed in the modern times is that the P (partitions) are extremely rare in modern high octane cloud infrastructures. Also, modern solutions often decide to sacrifice A (availability), by returning an error saying "we're aware of the problem and we're working on a solution". This is what happened quite recently when Google authentication went out and half of the internet went dark, while under the hood they had a simple out-of-quota situation on one of the replicas of their core authentication systems. The system was programmed to sacrifice A (availability) and reject all authentication requests.

Re:Why is IRC distributed across multiple servers? by moonchild on Sunday September 12, 2021 @04:13PM (#28504467)
> IRC has chosen to sacrifice C (consistency)

Hm? Hasn't it sacrificed partition tolerance? A netsplit is a partition.

Re:Why is IRC distributed across multiple servers? by Twisol on Sunday September 12, 2021 @04:59PM (#28504872)
It tolerates partitions just fine; I've been through many netsplits where folks just kept talking on our side of the split until the network healed.

Partition tolerance doesn't mean partitions don't affect the system, or that they can't happen. It just means the system has to choose whether to become unavailable or inconsistent (since it can't have both in the presence of a partition). IRC chooses to remain available, at the cost of losing messages for people on the wrong side of the split.

Re:Why is IRC distributed across multiple servers? by Doxin on Monday September 13, 2021 @02:58AM (#28508589)
IRC netsplits are a great example of what the "split brain" problem looks like from the inside.
Re:Why is IRC distributed across multiple servers? by mappu on Sunday September 12, 2021 @05:21PM (#28505094)
In CAP, the P happens whether you like it or not, and you get to choose between C-but-not-A or A-but-not-C.

IRC is an AP system. It stays up (+A) in a netsplit (+P) but the resulting servers are not consistent.

Re:Why is IRC distributed across multiple servers? by j56no on Sunday September 12, 2021 @04:16PM (#28504492)
If it had sacrificed P IRC would stop working in case of a net split. Instead it keeps working in an inconsistent state.
Re:Why is IRC distributed across multiple servers? by wlonkly on Sunday September 12, 2021 @06:08PM (#28505488)
No, "stop working" is Availability.
Re:Why is IRC distributed across multiple servers? by NovemberWhiskey on Sunday September 12, 2021 @10:04PM (#28507109)
As has been pointed out, in the real world, the CAP theorem comes down to "do I choose to offer service to everyone in the presence of a partition?"

Consider a replicated database. A replica that's partitioned from the others cannot receive or propagate updates and will become inconsistent with those on the other side of the partition.

If you allow that node to stay up, you're sacrificing consistency for availability; and you're "AP".

If you force that node down, you're sacrificing availability for consistency; and you're "CP".

In principle, you can choose "CA". That's equivalent to saying "I choose to offer no service at all in the presence of a partition", so that's kinda sorta strictly worse for most workloads than "CP" and therefore uncommon in practice.

Re:Why is IRC distributed across multiple servers? by throwaway20371 on Sunday September 12, 2021 @01:57PM (#28503287)
> "netsplits" exist in every distributed system, be it a chat app or a database, it's just the CAP theorem

Well let's not get carried away. Network partitions happen everywhere, but everything is not about the CAP theorem. CAP theorem is a very specific model that a lot of apps (even ACID databases) don't conform with. Comparing IRC to CAP theorem is like comparing it to ACID and saying, "IRC decided to sacrifice transaction integrity".

IRC didn't explicitly sacrifice the C in CAP, they designed a simple server protocol. They could have added a bunch of weirdness to hide splits from users, but it would have been unnecessarily complicated and not contributed significantly to the user experience.

Re:Why is IRC distributed across multiple servers? by H8crilA on Sunday September 12, 2021 @02:03PM (#28503343)
I'm sorry but I don't think you realise how simple and fundamental the CAP theorem is. It's almost a tautology. And yes it applies fully.

The most basic case is if there's absolutely no method of exchanging information from point A to point B. Then agents at A and B will not be able to communicate. That's it. Any system built to facilitate information exhange will either have to deliver incomplete information (C) or will have to refuse to operate (A).

Now then, as I said, nowadays it's extremely unlikely that there's truly no connection between any two major Internet hubs (though it can happen, hello BGP). It still happens in specific systems that do not work on any method of information transfer but rather on specific methods of information transfer. The IRC example requires specific servers to be up, not just a functioning IP routing between the end clients. If some server is not up then (at least temporarily) from IRC's point of view there's no way to deliver information from A to B. The Google auth outage example requires (among most likely many other things) disk space availability on specific servers for information exchange to happen.

Re:Why is IRC distributed across multiple servers? by TheDong on Sunday September 12, 2021 @02:35PM (#28503601)
> I don't think you realise how simple and fundamental the CAP theorem is

May I recommend reading "A Critique of the CAP Theorum - Martin Kleppmann", available as a PDF here https://arxiv.org/abs/1509.05393

As that paper points out, your definition of CAP theorem is simplified and incomplete to the point of being wrong, as many are.

As it also point out, CAP theorem doesn't really account for eventual consistency well.

I would argue that a chat protocol is a good place to perform eventual consistency, and those tradeoffs work well. During network partitions, have both sides of the partition continue to accept messages. Have the client mark messages with random unique IDs, and have each server mark messages with a server timestamp. The well-defined merge operation is now to sort by server-time and dedupe by message ID, such that if a message is sent to two servers it only displays once.

This doesn't work for IRC traditionally, since messages do not have unique IDs, and so no merge operation can deduplicate them, and servers do not store messages during netsplits (or at any time really), so they cannot be re-sent.

However, a similar system exists for other chat systems. matrix is a federated system of multiple servers, and when partitions occur, each server will still accept new messages, and later those messages will be made available to other servers and merged in at the appropriate time.

I think that CAP theorem's results are less interesting if you consider application-level resolutions to network issues (i.e. eventual consistency), and as I believe the paper also implies trotting it out constantly when talking about practical systems gets old fast.

Re:Why is IRC distributed across multiple servers? by H8crilA on Sunday September 12, 2021 @03:01PM (#28503818)
If you can always merge reordered edits/messages then CAP does not apply because you don't need C (as defined in CAP), you may instead talk about partitions/connectivity issues as if they were some anomalous sources of large latencies in the system. You have your own, different definition of C. There are some very very large scale systems out there that work under the assumption that any edits can arrive reordered, and it's OK for the observable properties of the system.

Here's what's "inconsistent" in an eventually consistent chat app: your typed responses might have been different had you seen in time what the other party has to say. To some degree the "computation" happens in your head. A "fully consistent" / "fully synchronous" chat app would sometimes refuse to send a message because the other party might have said something in the meantime. Like you'd expect from a fully-synchronous bank account balance handling system that wants to keep >= 0 balance at all times, rejecting overdraft transactions.

(And I agree that this is completely acceptable behavior for a chat app; we as people are built to tolerate this kind of a problem in async person to person communication; just pointing out what does C in CAP exactly mean; the "fully synchronous" chat app would be just an occasional pain in the ass with little benefit)

Re:Why is IRC distributed across multiple servers? by lmm on Sunday September 12, 2021 @09:11PM (#28506791)
> As that paper points out, your definition of CAP theorem is simplified and incomplete to the point of being wrong, as many are.

How so? Grandparent's statement conveys something true and useful, as far as I can see.

> As it also point out, CAP theorem doesn't really account for eventual consistency well.

It doesn't need to. CAP will tell you that some responses are inconsistent, and that remains true and important in an eventually consistent system.

> I would argue that a chat protocol is a good place to perform eventual consistency, and those tradeoffs work well. During network partitions, have both sides of the partition continue to accept messages. Have the client mark messages with random unique IDs, and have each server mark messages with a server timestamp. The well-defined merge operation is now to sort by server-time and dedupe by message ID, such that if a message is sent to two servers it only displays once.

Messages aren't the issue, things like operator permissions and channel takeovers are. It's not as simple as you're making it sound (e.g. a common problem for servers following your algorithm was that someone who got operator permissions on the wrong side of the split would correctly lose them on the merge, but bans that they'd created during the split would stay in place, allowing them to prevent the legitimate operators of a channel from exercising their control).

> I think that CAP theorem's results are less interesting if you consider application-level resolutions to network issues (i.e. eventual consistency), and as I believe the paper also implies trotting it out constantly when talking about practical systems gets old fast.

I too am frustrated that we have to trot it out so often, but for the opposite reason: the CAP theorem should be part of the common baseline that everyone understands, but even that much gets disputed. People constantly want to believe that their new technique (such as eventual consistency) has magically solved all their problems. The fact that the CAP theorem is extremely simplistic is a strength, as it cuts through a lot of obfuscating nonsense; in my experience people who want to dismiss it usually being naive about the impact that inconsistency will have on their system, just like in your IRC example above.

Re:Why is IRC distributed across multiple servers? by loo on Sunday September 12, 2021 @10:49PM (#28507360)
> As it also point out, CAP theorem doesn't really account for eventual consistency well.

Consistency is the first letter. Systems can be eventually available or eventually non-partitioned too.

You can promote two of the 'eventually's to 'now'. It ejects the other tautologically.

Re:Why is IRC distributed across multiple servers? by throwaway20371 on Sunday September 12, 2021 @02:11PM (#28503411)
> The most basic case is if there's absolutely no method of exchanging information from point A to point B. Then agents at A and B will not be able to communicate. That's it.

That's not it. The most basic case is if there's no linearizeability between A and B. A and B can continue communicating but fail the C in CAP if linearizeability fails. Hence we shouldn't compare everything to CAP.