leemoonsoo

Yes, if you already have k8s cluster that you don't manage, and you can use for your Spark, I would like to say you have Spark serverless environment.

IMO, where actual spark executor runs (either on lambda or k8s, or somewhere else) doesn't much matter to say it is serverless or not, while user can access the cluster, without managing them.

In our case, we build Spark Serverless service through

  - Fully managed Kubernetes cluster
  - Isolation between Spark instances (network, storage access, etc)
  - A dedicated Kubernetes node (VM) allocation for a executor, to provide container level security and better performance
  - Secure tunneling between remote Driver and executors for interactive mode (spark client deploy mode)
  - Various optimizations

When gVisor is enabled, hyper-threading is disabled (https://cloud.google.com/kubernetes-engine/docs/concepts/san...) and GKE cluster shows only half the CPU as available resources.

Does n2d instance is considered safe from CPU vulnerability and safely enable hyper-threading when used with gVisor?

I can spend hours and hours to explain why Zeppelin is good and what is different feature compare to the other.

for example, Zeppelin has pluggable architecture so it does support not only scala but also python with built in Spark integration. Zeppelin not only support data exchange between scala and python environment but also SparkContext sharing for spark cluster resource utilization. It's got ability create rich interactive analytics GUI inside of notebook. It's got customizable layout system, and so on..

But I wanted to address more fundamental difference. Most competitors are opensource project. And as a opensource project, how community works and project direction are the most important things. That'll make huge difference in the end. And i think i explained and compared them.

For someone who is interested, Here's some videos of Apache Zeppelin (Incubating)

https://www.youtube.com/watch?v=QdjZyOkcG_w

https://www.youtube.com/watch?v=_PQbVH_aO5E&feature=youtu.be

https://www.youtube.com/watch?v=cWuWUvWVLx4

https://www.youtube.com/watch?v=xU5TBS_MsAs

I'm one of committers of Apache Zeppelin (incubating). Zeppelin is inspired by iPython notebook and many other amazing softwares that has notebook interface.

I know iPython notebook has long history and large community, i really like it. Zeppelin is young, new project compare to iPython notebook.

Zeppelin and iPython notebook, they are opensource. iPython notebook is lead by IPython Development Team. Zeppelin is under Apache Software Foundation and it is being developed in Apache way, from copyrights, development process, decision making process to community development.

Zeppelin is focusing on providing analytical environment on top of Hadoop eco-system. I'm not sure about iPython's direction, but i don't think it's the same to Zeppelin.

I see many projects that has notebook interface. Not only iPython and Zeppelin, but also Databricks Cloud, Spark Notebook, Beaker and many others. I'm sure they all have their own advantages. Hope see all softwares are beloved by users.

Cheers