Ask HN: Does a dynamically scaling cloud workstation exist somewhere?

Re:Ask HN: Does a dynamically scaling cloud workstation exis... by tgdn on Wednesday October 13, 2021 @11:11AM (#28853179)

Have you looked into Spark? There are managed Spark options on AWS/GCP (for example Databricks). Spark lets you do exactly what you are saying.

Define minimum/maximum number of nodes, the machine capacity (RAM/CPU) and let Spark handle the scaling for you.

It gives you a Jupyter-like runtime to work on possibly massive datasets. Spark is perhaps too much for what you're looking for. Kubernetes could possibly be used with Airflow/DBT possibly, for example for ETL/ELT pipelines.

[ Parent | on HN ]

Re:Ask HN: Does a dynamically scaling cloud workstation exis... by ekns on Wednesday October 13, 2021 @01:33PM (#28855011)

Ideally I'd like to extend at least the illusion of an ad hoc PC/workstation to the cloud. For me it seems like it would be less effort until I reach some ridiculous scale that requires more engineering and setup anyway.

[ Parent | on HN ]