News for hackers. Stuff that matters.
Re:Revisiting 1M Writes per second by rsynnott on Saturday July 26, 2014 @12:06AM (#8088888)
> It is quite cheap to average a million writes per second. I've done it with five servers on AWS, and that was spatially indexing billions of GeoJSON polygons through storage while running queries against the index.

How big was your total dataset. It's cheap to average a million writes per second if the dataset fits in RAM, or at least if the index set does, with the right database. It can be less cheap for a data set far larger than RAM, as for most databases write amplification becomes a significant problem.

Re:Revisiting 1M Writes per second by jandrewrogers on Saturday July 26, 2014 @12:04PM (#8090006)
For that case, several terabytes IIRC. It was not in-memory, that database engine was pushing records through disk storage. Obviously there were no R-trees or any other kind of slow secondary indexing; the database itself is deeply and fundamentally spatially organized, even for text and numbers data.

While there is some write amplification it is less than most databases. It only takes few disks before the scheduler can get significantly more bandwidth out of the disks than a 10GbE network has to drive that activity, so there is extra capacity. The bottleneck on most server hardware is the silicon between the storage and memory if you are doing it right.