Re:Occupy the cloud: distributed computing for the 99%
by peterwwillis on Monday October 30, 2017 @12:20PM
(#15586236)
I re-read the article and now it makes more sense. The takeaway seems to be "serialize your functions in bulk and de-serialize later in bulk": in essence, treat your compute like your storage. But I don't see how this is distributed computing for the 99%?
PyWren appears to be what some of the serverless platform providers were missing, and also repeats a bit of history from the bad old days of parallel processing. While it totally works for a lot of scientific and a few business cases, it doesn't work for the general case. It's like saying MPI was the best solution for the 99%. But I might be overlooking another detail.
The biggest problem with depending on one aspect of a system for performance is that it becomes the bottleneck. We'll rely on low-latency, high-bandwidth storage? Now if it becomes low latency or low bandwidth, our app is boned. We'll rely on loads of cheap compute cycles? If compute nodes start choking, our app is boned. We'll rely on RAM? Our data becomes bigger, and our app is boned. We'll just scale horizontally? Our network gets partitioned, and our app is boned.
It can work fine for systems that don't require performance guarantees. If that's the 99%, then this seems like a good solution, though it also seems like you're trading off learning a complex stack for learning HPC development models.
PyWren appears to be what some of the serverless platform providers were missing, and also repeats a bit of history from the bad old days of parallel processing. While it totally works for a lot of scientific and a few business cases, it doesn't work for the general case. It's like saying MPI was the best solution for the 99%. But I might be overlooking another detail.
The biggest problem with depending on one aspect of a system for performance is that it becomes the bottleneck. We'll rely on low-latency, high-bandwidth storage? Now if it becomes low latency or low bandwidth, our app is boned. We'll rely on loads of cheap compute cycles? If compute nodes start choking, our app is boned. We'll rely on RAM? Our data becomes bigger, and our app is boned. We'll just scale horizontally? Our network gets partitioned, and our app is boned.
It can work fine for systems that don't require performance guarantees. If that's the 99%, then this seems like a good solution, though it also seems like you're trading off learning a complex stack for learning HPC development models.