Code sits on our laptops but live syncs to the remote services without requiring a Docker build or K8s deploy. It really does feel like local.
In particular it lets us do away with the commit-push-pray cycle because we can run integ tests and beyond as we code as opposed to waiting for CI.
We use Garden, (https://docs.garden.io) for this. (And yes I am afilliated :)).
But whether you use Garden or not, leveraging the power of the cloud for “inner loop” dev can be pretty amazing with right tooling.
I wrote a bit more about our experience here: https://thenewstack.io/one-year-of-remote-kubernetes-develop...
Many CI systems would spin up a new box instead of using persistent so likely have to rebuild if no cache, etc.
So basically I would say most of the overhead is in not having a persistent box with knowledge of last build or ability to choose what to run in there, which pretty much just equals to local capabilities.
Similarly if another dev (or a CI runner) triggers a build of one of our services, I won’t have to next time I start my dev environment. And because it’s built in the cloud there’s no “works on my machine”.
Same applies to tests actually. They run in the cloud in an independent and trusted environment and the results are cached and stored centrally.
Garden knows all the files and config that belong to a given test suite. So the very first CI run may run tests for service A, service B, and service C. I then write code that only changes service B, open a PR and only the relevant tests get run in CI.
And because it’s all in prod-like environments, I can run integ and e2e tests from my laptop as I code, instead of only having that set up for CI.
Admittedly, documentation and stability weren’t quite what we’d like and we’ve done a massive overhaul of the foundational pieces in the past 12 months.
If you want to share feedback I’m all ears, my email is in my profile.
There was an internal search provider (bunnylol) that had tools like putting @od in front of any FB URL to generate a redirect of that URL to your currently checked out On Demand server. Painless to work with! Nice side benefit of living on the same domain as the main sites is that the cookies are reused, so no need to log in again.
However, I also feel like our setup was well suited to remote-first dev anyway (eg. syncing of auto-generated files being a pain for local dev).
I work for a not-really-tech company (and I'm not a full-time dev either), so I've been issued a crappy "ultra-portable" laptop with an ultra-low-voltage CPU. I've looked into offloading my dev work to an AWS instance, but was quite surprised that it wasn't any faster than doing things locally for things like Rust compiles.
I’ve been integrating psrecord into our builds to track core utilisation during the built and see that a lot of time is spent in single threaded activities. Effort is required to compile modules in parallel but that is actually quite straightforward. Running all tests in parallel is harder.
We get the most out of the cloud machines by being able to provision a 16+ core machine to run more complicated (resilience) tests and benchmarks.
Also note that typically the cloud machines run on lower clocked CPUs than you would find in a workstation depending on which machine you provision.
This is honestly a bit overkill for a dev workstation (unless you compile Rust!), but since it’s a dedicated server it can also host any number of fully isolated services for homelab or saas. There’s nothing else like it in the wild, afaik.
We have interns coming in and fully ready within an hour or two of setup. Same way changing local machines is a breeze with very little downtime.
As you point out rebuilding small test datasets instead of just filtering the prod DB is an option, but those also need maintenance, and take a hell of time to make sure all the relevant cases are covered.
Basically, trying to flee from the bulk and complexity tends to bring a different set of hurdles and missing parts that have to be paid in time, maintenance and bugs only discovered in prod.
PS: the test DB is still reset everyday. Eorse thing happening is we need to do something else for a few hours until it's restored.
This sounds like the result of a company investing in tooling, rather than something specific to a remote dev env. Our local dev env takes 3 commands and less than 3 hours to go from a new laptop to a fully working dev env.
While my team (platform & infra) much prefer remote devbox, the development teams are not.
It could be specific to my org because we have way too many restrictions on the local dev machine (eg: no linux on laptop but it's ok on server and my team much prefer linux over crippled Windows laptop).
At large tech companies like Google, Meta, etc the dev environment is entirely in the cloud for the vast majority of SWEs.
This is a much nicer dev experience than anything local.
Can’t really say anything for performance, but I don’t think it’ll beat my laptop unless maven can magically take decent advantage of 32 cores (which I unfortunately know it can’t).
One thing that jumps out at me is the assumption that compile time implies wasted time. The linked Martin Fowler article provides justification for this, saying that longer feedback loops provide an opportunity to get distracted or leave a flow state while ex. checking email or getting coffee. The thing is, you don't have to go work on a completely unrelated task. The code is still in front of you and you can still be thinking about it, realizing there's yet another corner case you need to write a test for. Maybe you're not getting instant gratification, but surely a 2-minute compile time doesn't imply 2 whole minutes of wasted time.
More time working doesn't translate to being more effective and more productive. If that were the case then why are a disproportionate percentage of my "Oh shit! I know what to do to solve that..." in the shower, on my morning run, etc.?
I really struggle with task switching, and two minutes is the danger zone. Just enough time to get distracted, by something else; too little time to start meaningful work on anything else...
Hour long compiles are okay, I plan them, and have something else to do while they are building.
30 second compiles are annoying, but don't affect my productivity much (except when doing minor tweaks to UI or copywriting).
2-10 minute compiles are the worst.
"spend X time to automate this task vs not do the task at all".
"Spend X time to automate this task that takes Y×n time normally and get it down to Z×n time, vs spend Y×n time to do the task"Also not all minutes have equal value, spending a few hours to save 5 minutes in an emergency with automation can be well worth it.
It turns out that hooking every system call with vendor crapware is bad for a unix-style toolchain that execs a million subprocesses.
There’s moments where you try to do a thing that normal on a local PC and it’s impossible on remote. That cognitive dissonance is the worst.
When you need to have 200+ parts running to do anything, it can be hard to work in a single piece that touches a couple others.
With servers that have upwards of 128+ cores and 256+ threads, my opinion is swinging back in favor of monoliths for most software.
A word of warning from personal experience:
I am part of a medium-sized software company (2k employees). A few years ago, we wanted to improve dev productivity. Instead of going with new laptops, we decided to explore offloading the dev stack over to AWS boxes.
This turned out to be a multi-year project with a whole team of devs (~4) working on it full-time.
In hindsight, the tradeoff wasn't worth it. It's still way too difficult to remap a fully-local dev experience with one that's running in the cloud.
So yeah, upgrade your laptops instead.