> So you have LLM-based English prompts as an interop layer to Python + PySpark, which is itself an interop layer onto the Spark core.

And the spark core "just" generates execution plan, which on databricks gets executed as native code.

https://docs.databricks.com/runtime/photon.html