Re:The Infrastructure Behind Twitter: Scale
by seanp2k2 on Friday January 20, 2017 @04:59AM
(#13442683)
The flip side of that imo is that if you don't have Twitter-scale needs for the specific things they've optimized their infrastructure for, you probably don't need their solutions :)
Re:The Infrastructure Behind Twitter: Scale
by TheAceOfHearts on Monday January 23, 2017 @08:20PM
(#13468015)
I haven't used it, but one tool I've read good things about Minio [0]. I don't know if it's able to match twitter-scale, though.
Most of the time, we were building custom distribution layers on top of open source storage (e.g. memcached/mysql/redis/etc). I think blobstore was the first thing twitter put in production that was mostly custom, followed a year or two later by manhattan. I'm not sure if there's even now good open-source competitors for those projects, largely because any reasonable smaller company uses s3 or dynamo.
There's plenty of open-source things twitter created, or nurtured out of the existing ecosystem, from mesos to memcached to some of the hadoop/scalding/parquet stuff.