News for hackers. Stuff that matters.
Re:All of us test in production all the time (2019) by bluedino on Sunday July 19, 2020 @03:15PM (#23891637)
Most places that I've worked, that "test in prod", do so for one reason. It all comes down to laziness.

The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'.

The second scenario is that they do have a testing environment-however, for some reason, it's broken. Some change was made, usually in a hurry to meet a deadline, and it's been broken ever since. This one is usually a result of the fix being too difficult, because the test environment was haphazardly thrown together in the first place.

Re:All of us test in production all the time (2019) by sagichmal on Sunday July 19, 2020 @04:28PM (#23892205)
> The first scenario is they have never set up a test environment in the first place. They're either too lazy to do so, or too lazy to look into how to do it. Often confused with being 'too busy to do it'.

One lesson of modern architectures (i.e. anything more recent than the LiveJournal-style Web/App/DB 3-tier stack) is that it is literally impossible to create and maintain a test environment that has enough similarity to prod to be useful.

Re:All of us test in production all the time (2019) by UweSchmidt on Sunday July 19, 2020 @04:50PM (#23892326)
As QA I think testing, test environments, test ressources should be first class concepts in software. Building whatever "modern" architecture twice, and simulating traffic is not easy but possible, at least to an extent. You need to get that extra license for any piece of software for testing during procurement, and provide means to create test ressources, like typically test users as needed. Sounds trivial, but is often so complicated (banks, insurances...) that it should have been considered early during design.

So, I wouldn't use the word "lazy", but we could do better.

Re:All of us test in production all the time (2019) by virtue3 on Sunday July 19, 2020 @11:42PM (#23894479)
It's not the architecture, it's the representing all the states in prod in testing.

Not saying it's impossible, just not always feasible to have it actively "perfect".

I've found that an "ok" testing envrionment and solid unit testing / monitoring tends to do better.

And then b/g deploys and canaries :/

Re:All of us test in production all the time (2019) by sagichmal on Monday July 20, 2020 @08:47AM (#23897200)
Modern architectures look like this: https://github.com/donnemartin/system-design-primer

Building this out twice is either actually impossible, or so cost prohibitive as to be practically infeasible.

Re:All of us test in production all the time (2019) by eitland on Monday July 20, 2020 @10:04AM (#23897956)
> Modern architectures look like this:

A lot of modern architecture is not like that.

And even the diagram there is something should be possible to replicate if the organization values it.

Today it doesn't even need to be too expensive as it can be deployed with terraform and torn down an hour later when the full system tests are finished qnd go back to running the integration tests.

Re:All of us test in production all the time (2019) by sagichmal on Monday July 20, 2020 @11:39AM (#23898868)
It's not possible to replicate the CDN, or DNS, or the message queue if it's hosted, or the database of it's big enough, or etc. etc. The differences between what you're able to create in a staging environment and what exists in prod are significant enough that testing against the staging environment doesn't build much more confidence than testing with mocks.

Testing in prod means accepting this reality, avoiding the largely unproductive toil of building and maintaining a staging env, and making the production system observable, resilient, and operationally agile enough that you can deploy changes that can trigger unpredictable and emergent system behaviors while managing risk appropriately.

Re:All of us test in production all the time (2019) by hampfi on Monday July 20, 2020 @07:27AM (#23896751)
Not sure where you are working but not a lot of people work at/on a huge code base.

Most codebases are build and operated by 1-3 teams and if you are not able to reproduce your prod env as a test env, your architecture is broken.

Re:All of us test in production all the time (2019) by sagichmal on Monday July 20, 2020 @08:43AM (#23897169)
The size of the codebase is orthogonal to the size/complexity of the production environment.

Modern architectures look like this: https://github.com/donnemartin/system-design-primer

It is generally not feasible to recreate all of the elements of this design in a totally separate testing environment.

Re:All of us test in production all the time (2019) by hampfi on Monday July 20, 2020 @09:48AM (#23897773)
I know this very well and i'm stlil not seeing an issue.

We have 2 GCP accounts. One for production and one for test. Same Setup, not an issue due to terraform and the cost is reasonable for a test env.

Re:All of us test in production all the time (2019) by sagichmal on Monday July 20, 2020 @11:41AM (#23898894)
Same network rules and configuration? Same CDN settings? Same DNS? Same data in the databases? Same volume of traffic on the queues?

(Rhetorical questions. Of course the answer is no.)

All of these things have huge impact on the actual similarity of the systems, and consequently how they behave.

Re:All of us test in production all the time (2019) by hampfi on Tuesday July 21, 2020 @03:32AM (#23905014)
Obviously but this is not a black and white thing.

If it wouldn't make sense for us to have test and prod, we wouldn't do it but i can't imagine a scenario where this is NOT beneficial.

Re:All of us test in production all the time (2019) by Misdicorl on Sunday July 19, 2020 @05:23PM (#23892544)
To be useful? That seems to be an extraordinary low bar to hurdle for a QA environment. I imagine most clear that bar easily. I know mine does.
Re:All of us test in production all the time (2019) by baddox on Sunday July 19, 2020 @05:25PM (#23892562)
Can you explain that more? Aren’t most of the complex modern tools specifically designed to automate provisioning servers and deploying code?
Re:All of us test in production all the time (2019) by sagichmal on Monday July 20, 2020 @08:46AM (#23897191)
Modern architectures look like this: https://github.com/donnemartin/system-design-primer

All of the pieces have complex and emergent behaviors as they interact with each other. Many (most?) are hosted, not even fully in control of the team using them. Recreating this environment in a hermetic way for testing is either actually impossible, or so cost prohibitive as to be practically infeasible.

Re:All of us test in production all the time (2019) by isbvhodnvemrwvn on Sunday July 19, 2020 @06:57PM (#23893139)
I'm not the person you're responding to, but having the same software and hardware is possible (most of the time), but production environment is a living thing. To simulate production you need the same data and interaction you have in production, and that's quite difficult to do in many systems, if only for regulatory reasons (PII, PCI and friends).