jbs40

But any of those are infinitely better than naming something after a letter of the alphabet. That would be a crazy idea, wouldn't it?

(Sorry to all the users of 'R' and 'C' and 'S' and ...) :(

There seems to be a fundamental technical misunderstanding of these systems in the blog post. Unless you're running in the (generally quite dangerous) scenario where replication factor = 1, both EventBus and Kafka have a network hop (Kafka to the replica and EventBus to the storage layer) and JVM traversal, so the argument that Kafka is more efficient there doesn't make sense.

As you say, it would be trivial to run EventBus storage and serving on the same nodes if the network hop were an issue.

The argument that a decoupled storage layer costs more than co-locating storage and serving also seems dubious. The cluster is basically doing the same amount of work when you decouple those layers, it's just separating them into components that can be optimized for specific roles. So if you pick the right hardware/instances for each layer and tune for that, you really shouldn't need more hardware. Otherwise we'd all still be running full-stack applications on mainframes to minimize the number of servers. :)

I wonder if this would have built on Apache Pulsar (https://pulsar.apache.org) if it had been in open source and on the Segment team's radar at the time they started on Centrifuge. I work with one of the architects of Pulsar, and his first thought on seeing the blog was that Segment's scenario had a lot of similarities to what he and the team and Yahoo set out to do when they first built Pulsar there several years ago.

I was thinking more specifically of the internal architectures of data processing platforms, especially the categorizations that emerged from the MPP database world. The "shared nothing" architecture has been dominant in databases (and is also the core architecture of Hadoop), designed around "co-locating data and compute". Kafka largely follows that architecture as well, using local disk on the compute nodes as its persistent storage layer.

A lot of new data processing platforms, from Snowflake in the data warehouse world to AWS Athena to Apache Pulsar in the broader data processing world, have moved to decoupled architectures.

Containerization and container management frameworks (e.g. Kubernetes) certainly do change the meaning of "local" storage, will be interesting to see how that plays out.

The architectural pendulum is starting to swing away from co-location of storage and compute (the trend of the last 10+ years) to decoupling of storage and processing to avoid exactly these issues, but legacy architectures hang on for a while.

In the streaming and messaging space, Apache Pulsar (pulsar.apache.org) is a more recent solution that has an architecture that decouples processing and storage. That gives you nice properties like independent scaling of storage and processing, infinite data retention, dynamic resizing and others.

Forgot to state the disclaimer that I work for Snowflake.

Here's some useful updates and additional information on some of the items mentioned above regarding Snowflake:

- The concurrency limits mentioned above are soft limits that can be raised on customer request (those defaults are there so that runaway applications can be detected easily). Snowflake can handle very high concurrency--we have customers running hundreds of concurrent queries.

- We’ve recently released a new Spark connector with a bunch of optimizations, including additional push-down capabilities that speed up performance significantly.

- The clustering capability is currently in "preview", we're definitely taking input and have been working on incorporating feedback we've received so far into it.

- One important thing to note when it comes to full table scans is that Snowflake allows you to choose how much horsepower you apply to the job, so you can easily adjust the horsepower to get faster scans.