roshbhatia

In my experience, the case where this becomes really valuable is if your team needs access to either different kinds of hardware or really expensive hardware that changes relatively quickly (i.e. GPUs). At a previous small startup I setup https://devpod.sh/ (similar to gitpod) for our MLE/Data team. It was a big pro to leverage our existing k8s setup w/ little configuration needed to get these developer envs up and running as-needed, and we could piggyback off of our existing cost tracking tooling to measure usage, but I do feel like we already had infra conducive to running dev envs on k8s before making this decision -- we had cost tracking tooling, we had a dedicated k8s cluster for tooling, we had already been supporting GPU based workloads in k8s, and our platform team that managed all the k8s infra also were the SMEs for anything devenv releated. In a world where we started fresh and absolutely needed ephemeral devenvs, I think the native devcontainer functionality in vscode or something like github codespaces would have been our go to, but even then I'd push for a docker-compose based workflow prior to touching any of these other tools.

The rest of our eng team just did dev on their laptops though. I do think there was a level of batteries-included-ness that came with the ephemeral dev envs which our less technical data scientists appreciated, but the rest of our developers did not. Just my 2c

In my experience, it just worked -- I was at an org that ran 3rd party java services alongside our normal array of microservices (that all used our internal instrumentation library that wrapped OTEL) and using the OTEL autoinstrumentation for those 3rd party services was pretty trivial to get setup and running (just wrap the command to run the app with the OTEL wrapper, hand it a collector url.) Granted -- we already had invested in OTEL elsewhere and were familiar with many of the situations in which it didn't just work.

I'm with you here -- ECS deploys are pretty painless and uncomplicated, but I can picture a few scenarios where this ends up being necessary, for ex; if they have a lot of services deployed on ECS and it ends up bloating the size of the Terraform state. That'd slow down plans and applies significantly, which makes sharding the Terraform state by literally cloning the configuration based on a template a lot safer.