News for hackers. Stuff that matters.
Re:The Infrastructure Behind Twitter: Scale by fizx on Friday January 20, 2017 @02:50AM (#13442303)
I built custom storage for Twitter back in the 2010-12 period. There wasn't much off the shelf in those days that worked out-of-the-box at scale besides Cassandra, and Twitter had a well-documented attempt at using Cassandra for primary workloads that failed due to the amount of variance in IO and latency for high-volume workloads.

Most of the time, we were building custom distribution layers on top of open source storage (e.g. memcached/mysql/redis/etc). I think blobstore was the first thing twitter put in production that was mostly custom, followed a year or two later by manhattan. I'm not sure if there's even now good open-source competitors for those projects, largely because any reasonable smaller company uses s3 or dynamo.

There's plenty of open-source things twitter created, or nurtured out of the existing ecosystem, from mesos to memcached to some of the hadoop/scalding/parquet stuff.

Re:The Infrastructure Behind Twitter: Scale by seanp2k2 on Friday January 20, 2017 @04:59AM (#13442683)
The flip side of that imo is that if you don't have Twitter-scale needs for the specific things they've optimized their infrastructure for, you probably don't need their solutions :)
Re:The Infrastructure Behind Twitter: Scale by TheAceOfHearts on Monday January 23, 2017 @08:20PM (#13468015)
I haven't used it, but one tool I've read good things about Minio [0]. I don't know if it's able to match twitter-scale, though.

https://minio.io/