News for hackers. Stuff that matters.
Re:Hello World on z/OS by bunderbunder on Tuesday July 31, 2018 @08:55AM (#17652739)
FWIW, you could do the same thing with a Spark cluster. It might not handle it quite as gracefully, since you'd be taking down entire compute nodes, and probably some of the data storage along with them, but the computation would also be able to continue.
Re:Hello World on z/OS by dralley on Tuesday July 31, 2018 @09:01AM (#17652775)
Yes, but I'm guessing this predates Spark by decades. Not sure of the exact timeframe, though.

In any case, it helps explain why a purse-string-holding exec would be convinced to shell out money to IBM.

Re:Hello World on z/OS by fs111 on Tuesday July 31, 2018 @10:36AM (#17653482)
spark clusters still have a spof as far as I understand the architecture.
Re:Hello World on z/OS by bunderbunder on Tuesday July 31, 2018 @11:27AM (#17653940)
Two of them, I believe - the master for the cluster as a whole, and, for any individual job, its driver.

I wasn't really meaning to say that a Spark cluster is as robust as a mainframe (though maybe someone's figured out some tricks), more that I could totally see some sales folks conducting the same demo using something like Spark.

Re:Hello World on z/OS by shub on Tuesday July 31, 2018 @07:05PM (#17658096)
If you're using Spark's built-in scheduler then the cluster manager is a SPOF. Hadoop docs say you can get active/standy ResourceManager, not that I've tried it. Spark can also use k8s and nomad to schedule executors, and those have HA modes as well. I assume Mesos does HA.

You're still boned if the driver dies. I am pretty sure that the driver keeps some important state in RAM so if the node hosting it goes down you have to restart from the beginning, even if the cluster manager restarts the driver.