News for hackers. Stuff that matters.
Re:Stream and Go: News feeds for 300M users, built on RocksD... by jnordwick on Saturday January 20, 2018 @03:33AM (#16192614)
Let's do some math and take their metrics and divide by their number of servers:

- 6.3 million feed updates per day per server

- 111 API requests per minute per server

If you guess 4 cores per server it even sounds worse.

(Ps, will HN ever support better formatting?)

Re:Stream and Go: News feeds for 300M users, built on RocksD... by latch on Saturday January 20, 2018 @03:59AM (#16192675)
They measured their response time using average. Collecting metrics to look good.

Since we can only speak in broad terms, those requests/updates aren't going to be evenly distributed throughout the day and they might not be consistent from day to day so some over-provisioning is reasonable.

Since they're on EC2 a) they shouldn't be that over-provisioned though and b) are getting comparatively (to bare metal) awful performance (say 2x-8x depending on exactly what they're doing) and paying for the privilege.

I'd like to better understand why "Fanout-on-write" wasn't their go-to solution. All they said was that it was "expensive".

Re:Stream and Go: News feeds for 300M users, built on RocksD... by tschellenbach on Saturday January 20, 2018 @11:23AM (#16193878)
We started out with fanout on write but it was tricky. Fanout on write can get incredibly expensive. Customers would implement things like featured accounts, or topics that you can follow. Basically feeds that are followed by a significant percentage of your userbase and update all the time. Companies like Twitter and Instagram control their product and can avoid some of these pitfalls. A full fanout on write approach means you need to enforce certain product limitations, have a large budget and accept that feed propagation is not always as fast.
Re:Stream and Go: News feeds for 300M users, built on RocksD... by xstartup on Sunday January 21, 2018 @08:38AM (#16197983)
How does fan out write help?
Re:Stream and Go: News feeds for 300M users, built on RocksD... by latch on Monday January 22, 2018 @02:40AM (#16202580)
Lets you shard and makes your reads dead simple (1 table, 1 predicate, 1 sort)
Re:Stream and Go: News feeds for 300M users, built on RocksD... by tbarbugli on Saturday January 20, 2018 @11:30AM (#16193924)
There are more factors to take into account than plain API / feed update traffic when looking at server count:

- High Availability requires 2x or 3x more servers

- We run our own monitoring, log collection and tracing infrastructure

- ML/Analytics features

- Distribution of traffic

- Disaster Recovery

Edit: fix list formatting