Re:Ask HN: Should I quit the field of software development?
by asdfasgasdgasdg on Saturday September 19, 2020 @11:52AM
(#24528272)
I don't think so. Maybe for the active users, but maybe not even then. Most twitter users are inactive and so you wouldn't want to fan that kind of data out so widely. You'd either have to pay enormous gc costs or enormous storage costs for someone like Donald Trump (0.5 GB per tweet assuming 64 bit tweet IDs). But I think I would just have a sharded in memory cache of recent tweets for the big dogs and if someone loads their feed you would do lookups into the sharded db for each subscription. You'd then cache the feed for the user so you don't have to do that lookup very often.
If you were going to write, you'd write into the cache, rather than persisting anything to disk. That's my guess at least.
Seems to me you’d need to do the latter right? That way you ensure you process each follower at least once, but each worker doesn’t need to be aware of others it’s just pulling messages from the queue.