I don't understand your point.

The type of companies who could afford this are the types of companies who need 1M writes/second. Which are few and far between. And yes 10K is not that impressive but 1M is. And with Cassandra you could continue to improve that number just be rolling out more nodes.

It is quite cheap to average a million writes per second. I've done it with five servers on AWS, and that was spatially indexing billions of GeoJSON polygons through storage while running queries against the index.

Many companies need far in excess of a million writes per second. Basically, most machine-generated data sources, whether it is personal location data or any other kind of telemetry. Many companies that do not generate that data themselves buy and consume it. I know of companies doing over a billion writes per second.

Cassandra is pretty good for this type of thing among open source software but it is not nearly as efficient as it could be in terms of write throughput. If the storage engine is correctly designed, you should be able to drive 10GbE all the way through storage -- call it 1 GB/sec per node. However, that does mean you can't do things like mmap()-ing files; those interfaces are slow due to poor scheduling by the OS when the throughput is very high.

> It is quite cheap to average a million writes per second. I've done it with five servers on AWS, and that was spatially indexing billions of GeoJSON polygons through storage while running queries against the index.

How big was your total dataset. It's cheap to average a million writes per second if the dataset fits in RAM, or at least if the index set does, with the right database. It can be less cheap for a data set far larger than RAM, as for most databases write amplification becomes a significant problem.

For that case, several terabytes IIRC. It was not in-memory, that database engine was pushing records through disk storage. Obviously there were no R-trees or any other kind of slow secondary indexing; the database itself is deeply and fundamentally spatially organized, even for text and numbers data.

While there is some write amplification it is less than most databases. It only takes few disks before the scheduler can get significantly more bandwidth out of the disks than a 10GbE network has to drive that activity, so there is extra capacity. The bottleneck on most server hardware is the silicon between the storage and memory if you are doing it right.

Out of curiosity, what would a correctly designed storage engine do to get better throughput than mmap()ing files?

On Linux, you would use io_submit + O_DIRECT on a small number of large files allocated in large chunks. In short, you become the I/O scheduler and buffer manager instead of the operating system, while removing most of the implicit context switches. It does require much more code than mmap()-ing and fairly sophisticated code at that.

If you are doing it well, 3-5x throughput improvement seems to be average upside in my experience, which is huge. The scheduler behind mmap() simply does not have enough context about the workload to make good paging decisions leading to a lot of suboptimal or wasted I/O, and this is magnified when the storage I/O is under pressure. In principle, if you write your own I/O scheduler you can always make sure that the optimal I/O operation is executed at the optimal time.

They probably mean 10K writes/sec/node, which would be correct assuming a replication factor of three. It doesn't sound huge, but if it were sustained, with a key set vastly larger than memory, it's not too bad.

How would you build out at a metrics cluster that needs 1M writes/sec?

285 nodes that are easily automated to create/destroy/monitor doesn't seem like a management pain to me, personally. Just depends if you have the write internal tools built.

I have seen single servers do 250k writes/second. So it really depends on the data and access patterns more than some arbitrary one size fit's all solution.

I suspect systems where the dataset, or at least the indexes, largely fit in memory?

Most of the active data-set fit in ram, but considering you can get 100+GB of RAM that's generally not much of an issue. Also a single mid range SSD can break 120K writes per second so 0+1 RAID arrays can get crazy fast for less money than you might think.

Well, I mean, you _can_ get 100+GB of RAM, but you'll certainly pay for it. The machines used in the example have 30GB of RAM and 800GB SSD. For a fairly random access pattern and most of the storage used, the RAM's not going to be helping that much. Let's say they're using 600GB of the 800GB, replication factor of three, that's 57TB over their cluster. It's pretty big.

Uhh, you can get 128GB on a basic dell server that costs less than 4k. It's 512+GB of ram that starts getting pricey now days and 1TB is actually an option.

Though, I agree if you dataset is large enough and you need random access it's not going to help much.

Ok, so you're super-admin. And about the ROI aspect? What's the cost for achieving the 1M write/sec metric in that mammoth cluster?

I asked you how you would architect something that needs 1M writes/sec if you weren't using Cassandra. Ignore the fact that netflix is making this into a single cluster.

I'm not saying I'd do it by myself. I'm saying with the write tools its doable and not unreasonable.

The replication factor is set to 3, meaning that all data is stored on tree separate nodes - in different availability zones. So in practice the writes per sec is 3x.

Well, it's not unreasonable (the maintenance part) with Cassandra.