News for hackers. Stuff that matters.
Re:Staging is dead: The rise of preview environments by echelon on Thursday December 22, 2022 @12:42PM (#34096080)
I have spent a lot of time thinking about this problem. I would gladly pay for a product that could make a real, honest to god preview environment that you could spin up on-demand and populate it with all of the data you need.

This isn't it, and I think it's pretty much an impossible problem to solve without an entirely new engineering culture around testing.

How do you duplicate the behavior of the 10 direct data dependencies of your app and the constellation of transitive microservice dependencies? Wingman, Galactus, etc. At a company with 1000+ microservices?

And how do you populate it with actual data so you can log in and perform actions? Every single microservice probably needs test data, and it would take a lot of engineering forethought to populate the entire graph.

It gets gnarly. I've been on teams at the center of it all - user accounts and login, representation of the core business entities that the rest of the company is built upon. We couldn't even solve the problem for ourselves. There were so many permutations of account states, creating a dedicated API to make test data deviates from a first class API. So how do you coordinate with downstream services to skip IDV (matching SSN), compromised password checks, etc.

Perhaps you only bring online a subset of services and fake out the rest. At what level can you fake stuff out? And how do you ensure that the test data works with whatever subset you don't fake? If a single service responds in an unexpected way, it could break or corrupt the state you're relying upon to test.

Maybe "preview" works for frontend. Backend, not so much. This is such a massive problem, and I'm only scratching the surface with my description of the kinds of issues you have to solve.

Re:Staging is dead: The rise of preview environments by hangonhn on Thursday December 22, 2022 @01:39PM (#34096773)
That's a really good point.

I think what made it quite easy for us is that we maintain a staging environment. Staging is constantly tested by QA simulating our workflows so non-production data is being constantly generated. When we spin up our preview environments, we use snapshots from the staging systems to seed the preview environments.

Your point on data though really gets to the crux of the problem. Deploying the services is trivial these days with k8s, Cloud Formation, Terraform, Docker, etc. but the data part is hard.

Re:Staging is dead: The rise of preview environments by jcraft on Thursday December 22, 2022 @02:28PM (#34097407)
> but the data part is hard.

I'd love to learn more about the challenges you have with data. My company essentially provides data-anonymization-as-a-service and we integrate with ephemeral environments, but we could probably improve the integrations.

Re:Staging is dead: The rise of preview environments by hangonhn on Thursday December 22, 2022 @06:22PM (#34100321)
Got a link to your company's service? Would love to learn more. We work in healthcare so data-anonymization is essential. If we can take prod data and erase all the PII, it would be AMAZING.
Re:Staging is dead: The rise of preview environments by phendrenad2 on Thursday December 22, 2022 @12:47PM (#34096141)
Microservices became this mess because people only think about production. Nobody builds microservices as standalone units that should be testable and runnable in isolation, with ZERO external data. Instead, everything relies on everything, so it's just a monolith split up into microrepos. Worse, people have started putting app configuration in S3 and other dumb things (I've seen it...) do you cannot run things locally, or even match production unless you have AWS creds. Madness.
Re:Staging is dead: The rise of preview environments by TimTheTinker on Thursday December 22, 2022 @01:14PM (#34096484)
Microservices should indeed be able to run in isolation, but that doesn't remove the need for end-to-end and integration testing.

At a minimum, the product owner should be able to try out the new feature in the UI without having to deal with a staging environment.

Re:Staging is dead: The rise of preview environments by gedy on Thursday December 22, 2022 @01:55PM (#34096974)
> Nobody builds microservices as standalone units that should be testable and runnable in isolation, with ZERO external data. Instead, everything relies on everything, so it's just a monolith split up into microrepos.

Yeah this is the surprising thing to me - I've worked with otherwise bright people but whenever we've introduced services/microservices, it seems the default reaction is to just make a "distributed monolith" with circular dependencies and tight coupling.

I'm only half joking that 90% of my "architect" role is just point out we shouldn't have circular dependencies.

Re:Staging is dead: The rise of preview environments by codethief on Thursday December 22, 2022 @02:29PM (#34097419)
> Instead, everything relies on everything, so it's just a monolith split up into microrepos

…and with added network calls!

> Microservices. grug wonder why big brain take hardest problem, factoring system correctly, and introduce network call too. seem very confusing to grug

(Source: https://grugbrain.dev/)

Re:Staging is dead: The rise of preview environments by jeremy_k on Thursday December 22, 2022 @01:14PM (#34096487)
Sounds like a large dependency graph and multiple variants of spinning up the environments.

I'll try to simplify what I mean by that but something akin to having Services A, B, and C.

Each of A, B, and C need their own definitions to define how they need to be spun up as well as a definition on how to populate their data. That data population, say through RDS, needs to be spin up RDS instances and make them ready and available whenever someone creates a corresponding ephemeral version of a service. Toss the database afterwards.

In the above case, assuming you needed a full graph, you would spin up the three services as well as attach each service to its corresponding database. This approach works well when the number of services remains small.

Once that number starts growing, it may not be cost effective to spin everything up. Say service N is needed, but you really only need to read from it. In this case, you might have a dedicated staging environment for N and create a dependency graph that says "in my ephemeral environment, create new instances of A, B, and C with their databases, but also allow network requests to make their way to N."

That might be over simplifying, but I've been working in this space for the past few years and have been thinking about this exact issue recently

Re:Staging is dead: The rise of preview environments by robertlagrant on Thursday December 22, 2022 @12:54PM (#34096226)
The way I originally did it at my previous company was to have each service have a "dev" Flask blueprint that got registered on non-production environments, that had an endpoint that could populate the database with sample data, and a coordinating service to populate them in the right order (so IDs matched up). Worked pretty well.
Re:Staging is dead: The rise of preview environments by howinteresting on Thursday December 22, 2022 @01:18PM (#34096549)
While the problem is still hard, it is definitely easier to solve with fewer, larger services. If your backend consists of 3-5 services, the problem is far more tractable than if it consists of 1000+ services.
Re:Staging is dead: The rise of preview environments by hangonhn on Thursday December 22, 2022 @01:37PM (#34096749)
Yeah that's a really good point. Our microservices are like 30? 1000+ is intractable but up to 50 or even a 100 it's doable. We rely pretty heavily on AWS Cloudformation to deploy. All our services have Cloudformation templates. I'm sure Terraform would work too or be even better. Since every service knows how to deploy itself, building a preview environment wasn't too difficult. It's just a matter of stitching them all together, which was made a lot easier with Cloudformation nesting of stacks.
Re:Staging is dead: The rise of preview environments by codethief on Thursday December 22, 2022 @02:25PM (#34097358)
> How do you duplicate the behavior of the 10 direct data dependencies of your app and the constellation of transitive microservice dependencies? Wingman, Galactus, etc.

For those, like me, who had to think for a moment where they had heard that reference: https://www.youtube.com/watch?v=y8OnoxKotPQ