News for hackers. Stuff that matters.
Re:Mudge Twitter whistleblower testimony [video] by seafoam on Tuesday September 13, 2022 @10:42AM (#32824899)
mudge just testified in response to a Senator's question that Twitter does not have a test env
Re:Mudge Twitter whistleblower testimony [video] by bushbaba on Tuesday September 13, 2022 @10:53AM (#32825043)
To be fair. Many companies staging is unusable with testing in prod.
Re:Mudge Twitter whistleblower testimony [video] by nrmitchi on Tuesday September 13, 2022 @10:58AM (#32825135)
If I recall correctly, Uber literally did an entire post series about how they do all testing in production using feature flags and request routing, because it was "technically infeasible" at their scale to do anything differently.
Re:Mudge Twitter whistleblower testimony [video] by m00x on Tuesday September 13, 2022 @04:53PM (#32830042)
Uber really isn't a great company to aspire to when it comes to tech. They have a similar mentality to Facebook of: "Spend more time getting it done than planning it out".
Re:Mudge Twitter whistleblower testimony [video] by smt88 on Tuesday September 13, 2022 @11:04AM (#32825234)
What does that mean though?

Twitter must have test environments for developers and automatic tests.

Do they just not have a full staging environment? That would be understandable. It's impossible to create a production-like test environment for a product like Twitter. You're better off just doing canary releases or feature flags with slow rollouts.

Re:Mudge Twitter whistleblower testimony [video] by shadowgovt on Tuesday September 13, 2022 @11:50AM (#32825902)
> What does that mean though?

"Every company has a test environment. Some companies are disciplined enough to have a production environment separate from the test environment."

Re:Mudge Twitter whistleblower testimony [video] by csours on Tuesday September 13, 2022 @03:09PM (#32828884)
Everyone tests in prod. Some also test before prod.
Re:Mudge Twitter whistleblower testimony [video] by michaelt on Tuesday September 13, 2022 @11:30AM (#32825606)
> It's impossible to create a production-like test environment for a product like Twitter.

What makes you think that?

I mean, I know it costs money and needs people who understand how the product works and how customers behave - but why on earth would it be impossible?

Re:Mudge Twitter whistleblower testimony [video] by smt88 on Tuesday September 13, 2022 @12:39PM (#32826663)
It's impossible because you need to simulate the volume and variety of Twitter usage. Especially when you consider that production Twitter is thousands of servers, you're just not going to be able to create a mirror of production.

For something with many millions of users, it makes more sense to do canary releases to internal users, then to a few thousand, and then scale up from there.

Re:Mudge Twitter whistleblower testimony [video] by TechBro8615 on Tuesday September 13, 2022 @01:13PM (#32827220)
You don’t need one staging environment simulating the entire platform. You can have small staging environments that run some parts of the product, for example the latest version of the advertiser dashboard or the messaging inbox.

Sure, you won’t be simulating a whole class of problems that can arise at scale, but you can still test the more localized features in a safe environment.

Re:Mudge Twitter whistleblower testimony [video] by smt88 on Tuesday September 13, 2022 @03:55PM (#32829432)
Yes, I understand that. That's why it's unclear what it means that Twitter has "no test environment". That could mean a lot of things, some catastrophic and some not.
Re:Mudge Twitter whistleblower testimony [video] by m00x on Tuesday September 13, 2022 @04:55PM (#32830063)
It's pretty simple. They have no testing environment. They yeet shit in prod under feature flags (I assume), and hope for the best.
Re:Mudge Twitter whistleblower testimony [video] by michaelt on Tuesday September 13, 2022 @01:11PM (#32827190)
If you can accept, index and store a given volume of tweets, why should it be impossibly difficult to generate that same volume? I would imagine it would be a lot easier, in fact.

And if can't afford a 1:1 scale replica, build a 1/10th scale replica and have your senior engineers use their expertise to ensure it's representative enough for test purposes.

Re:Mudge Twitter whistleblower testimony [video] by sjsdaiuasgdia on Tuesday September 13, 2022 @03:33PM (#32829177)
It's a much harder target to generate truly representative load. Sure, you could make millions of tweets very fast, but is that actually representing what the production service is dealing with?

Unstable connections. Traffic spikes that can be highly localized or nearly system wide. System maintenance, planned and reactive. Poorly behaved clients. Malicious traffic. Many more similar factors...and yes, each of these you could throw engineers at them and get solutions eventually. You might also end up with a testing engineering team that dwarfs the product engineering team.

Also, Twitter doesn't make money. $5B revenue vs $5.5B opex in 2021. They can't really afford to run the actual system, much less a fully representative one. Even a 1/10th replica, which would be unable to find some classes of problems no matter how many engineers you throw at, would further break the bank. As would the engineering staff to support it.

Re:Mudge Twitter whistleblower testimony [video] by smt88 on Tuesday September 13, 2022 @03:58PM (#32829462)
Twitter can't afford a 1:1 replica for sure, and it would be meaningless without globally distributed production load anyway.

A 1:10 replica is fine, but also astronomically expensive. What's better about it than 1:100? Or 1:10000?

Once you give up on 1:1 (which is reasonable), you're either testing in production or not really testing at all. You'll have many lines of code that will only apply to the 1:1 scale.

Re:Mudge Twitter whistleblower testimony [video] by bragr on Tuesday September 13, 2022 @12:46PM (#32826770)
>What makes you think that?

For large scale sites you need a parallel copy of the internet which doesn't exist. For example, we once caused a production incident (for us and other people) by simulating a major business event to our test stack and saturated links between <Cloud Provider> and <CDN Provider>. For future tests that was always an edge case that we couldn't fully test. Rinse and repeat those kinds of problems with every vender, service, and traffic source.

Re:Mudge Twitter whistleblower testimony [video] by faeriechangling on Tuesday September 13, 2022 @08:55PM (#32832001)
The amount of companies that have a REAL test environment is far smaller than the ones with a nominal test environment as well.
Re:Mudge Twitter whistleblower testimony [video] by bombcar on Tuesday September 13, 2022 @10:51AM (#32825017)
This is clearly perjury, you can find it at www.twitter.com

/s