We Don’t Use Docker

Re:We Don’t Use Docker by trashtester on Tuesday March 16, 2021 @06:53AM (#26475116)

My rule of thumb is that a single processor core can handle about 100MB/s, if using the right software (and using the software right). For simple tasks, this kan be 200+ MB/s, if there is a lot of random access (both against memory and against storage), one can assume about 10k-100k IOPS per core.

For a 32 core processor, that means that it can process a data set of 100G in the order of 30 seconds. For some types of tasks, it can be slower, and if the processing is either light or something that lets you leverage specialized hardware (such as a GPU), it can be much faster. But if you start to take hours to process a dataset of this size (and you are not doing some kind of heavy math), you may want to look at your software stack before starting to scale out. Not only to save on hardware resources, but also because it may require less of your time to optimize a single node than to manage a cluster.

[ Parent | on HN ]

Re:We Don’t Use Docker by xnx on Tuesday March 16, 2021 @07:54AM (#26475539)

> "using the right software (and using the software right)"

This is a great phrase that I'm going to use more.

[ Parent | on HN ]

Re:We Don’t Use Docker by yowlingcat on Tuesday March 16, 2021 @09:14AM (#26476249)

This is a great rule of thumb which helps build a kind of intuition around performance I always try to have my engineers contextualizing. The "lazy and good" way (which has worked I'd say at least 9/10 times in my career when I run into these problems) is to find a way to reduce data cardinality ahead of intense computation. It's 100% for the reason you describe in your last sentence -- it doesn't just save on hardware resources, but it potentially precludes any timespace complexity bottlenecks from becoming your pain point.

[ Parent | on HN ]