News for hackers. Stuff that matters.
Re:Netflix buffering issues: Boxing fans complain about Jake... by softwaredoug on Saturday November 16, 2024 @12:53PM (#42157774)
The way to deal with this is to constantly do live events, and actually build organizational muscle. Not these massive one off events in an area the tech team has no experience in.
Re:Netflix buffering issues: Boxing fans complain about Jake... by mbrumlow on Saturday November 16, 2024 @02:56PM (#42158774)
I have this argument a lot in tech.

We should always be doing (the thing we want to do)

Somme examples that always get me in trouble (or at least big heated conversations)

1. Always be building: It does not matter if code was not changed, or there has been no PRs or whatever, build it. Something in your org or infra has likely changed. My argument is "I would rather have a build failure on software that is already released, than software I need to release".

2. Always be releasing: As before it does not matter if nothing changed, push out a release. Stress the system and make it go through the motions. I can't tell you how many times I have seen things fail to deploy simply because they have not attempted to do so in some long period of time.

There are more just don't have time to go into them. The point is if "you did it, and need to do it again ever in the future, then you need to continuously do it"

Re:Netflix buffering issues: Boxing fans complain about Jake... by andai on Saturday November 16, 2024 @03:12PM (#42158896)
This is great, but what possible counterargument is there? We should prolong indefinitely a spooky ambiguity about whether the system works or not?
Re:Netflix buffering issues: Boxing fans complain about Jake... by jerf on Saturday November 16, 2024 @03:57PM (#42159259)
Easy: Short term risk versus long term risk. If I deploy with minimal changes today, I'm taking a non-zero short-term risk for zero short-term gain.

While I too am generally a long-term sort of engineer, it's important to understand that this is a valid argument on its own terms, so you don't try to counter it with just "piffle, that's stupid". It's not stupid. It can be shortsighted, it leads to a slippery slope where every day you make that decision it is harder to release next time, and there's a lot of corpses at the bottom of that slope, but it isn't stupid. Sometimes it is even correct, for instance, if the system's getting deprecated away anyhow why take any risk?

And there is some opportunity cost, too. No matter how slick the release, it isn't ever free. Even if it's all 100% automated it's still going to barf sometimes and require attention that not making a new release would not have. You could be doing something else with that time.

Re:Netflix buffering issues: Boxing fans complain about Jake... by ukuina on Saturday November 16, 2024 @03:18PM (#42158962)
Finite compute, people, and opportunity cost.

It is just a reframing of build vs maintain.

Re:Netflix buffering issues: Boxing fans complain about Jake... by macintux on Saturday November 16, 2024 @03:33PM (#42159076)
In some environments, deploying to production has a massive bureaucracy tax. Paperwork, approvals, limited windows in time, can’t do them during normal business hours, etc.
Re:Netflix buffering issues: Boxing fans complain about Jake... by lelandfe on Saturday November 16, 2024 @03:49PM (#42159187)
I worked somewhere where a deployment took 14hrs and left the site down for the last hour. Failure required a laborious DB rollback, which happened semi-regularly. (Magento 2 IYKYK)
Re:Netflix buffering issues: Boxing fans complain about Jake... by josho on Saturday November 16, 2024 @05:18PM (#42159878)
Those taxes were often imposed because of past engineering errors. For example, Don't deploy during business hours because a past deployment took down production for a day.

A great engineering team will identify a tax they dislike and work to remove it. Using the same example, that means improving the success rate of deployments so you have the data (the success record) to take to leadership to change the policy and remove the tax.

Re:Netflix buffering issues: Boxing fans complain about Jake... by mplewis on Saturday November 16, 2024 @03:16PM (#42158935)
The common and flawed counterargument is “when we deploy, outages happen.” You’ll hear this constantly at companies with bad habits.
Re:Netflix buffering issues: Boxing fans complain about Jake... by kortilla on Saturday November 16, 2024 @03:55PM (#42159241)
Deploying is expensive for some models. That could involve customer facing written release notes, etc. Sometimes the software has to be certified by a govt authority.

Additionally, refactor circle jerks are terrible for back-porting subsequent bug fixes that need to be cherry picked to stable branches.

A lot of of the world isn’t CD and constant releases are super expensive.

Re:Netflix buffering issues: Boxing fans complain about Jake... by rconti on Saturday November 16, 2024 @04:46PM (#42159634)
The counterargument is obvious for anyone who has been on call or otherwise responsible for system stability. It's very easy to become risk-averse in any realm.
Re:Netflix buffering issues: Boxing fans complain about Jake... by andai on Saturday November 16, 2024 @06:27PM (#42160484)
Doesn't ensuring stuff actually works tangibly lower risk?
Re:Netflix buffering issues: Boxing fans complain about Jake... by Jach on Saturday November 16, 2024 @10:34PM (#42161786)
Not exactly, but it's worth the experiment in trying things anyway. Say you currently have a release once every few months, an ambitious goal would be to get to weekly releases. Continuous enough by comparison. But 'lower risk' is probably not the leading argument for the change, especially if the quarterly cycle has worked well enough, and the transition itself increases risk for a while. In order for a committed attempt to not devolve into a total dumpster fire, various other practices will need to be added, removed, or changed. (For example, devs might learn the concept of feature flags.) The individuals, which include management, might not be able to pull it off.
Re:Netflix buffering issues: Boxing fans complain about Jake... by mbrumlow on Saturday November 16, 2024 @10:40PM (#42161813)
Yes. Because it lowers the chance compound risk. The longer you go without stressing the system the more likely you will have a double failure, thus increasing your outage duration.

Simply put. You don’t want to delay funding out something is broke, you want to know the second it is broken.

The the case I am suggesting, a failed release will be often deploying the same functionality, thus many failure modes will result in zero outage. It all failure modes will result in an outage.

When the software is expected to behave differently after the deployment, more systems can result in being part of the outage. Such as the new systems can’t do something or the old systems can’t do something.

Re:Netflix buffering issues: Boxing fans complain about Jake... by parasti on Saturday November 16, 2024 @03:00PM (#42158807)
This is golden advice, honestly. "If you don't use it, you lose it" applied to software development.
Re:Netflix buffering issues: Boxing fans complain about Jake... by skybrian on Saturday November 16, 2024 @05:08PM (#42159793)
Doing dry runs regularly makes sense, but whether actually shipping it makes sense seems context-dependent. It depends on how much you can minimize the side effects of shipping a release.

Consider publishing a new version of a library: you'd be bumping the version number all the time and invalidating caches, causing downstream rebuilds, for little reason. Or if clients are lazy about updating, any two clients would be unlikely to have the same version.

Or consider the case when shipping results in a software update: millions of customer client boxes wasting bandwidth downloading new releases and restarting for no reason.

Even for a web app, you are probably invalidating caches, resulting in slow page loads.

With enough work, you could probably minimize these side effects, so that releasing a new version that doesn't actually change anything is a non-event. But if you don't invalidate the caches, you're not really doing a full rebuild.

So it seems like there's a tension between doing more end-to-end testing and performance? Implementing a bunch of cache levels and then not using it seems counterproductive.

Re:Netflix buffering issues: Boxing fans complain about Jake... by bonestamp2 on Saturday November 16, 2024 @06:03PM (#42160284)
I like all of these considerations, although I also imagine for every context there is some frequency at which it is worthwhile to invalidate the caches to ensure that all parts of the system are still functioning as expected (including the rebuilding of the caches).
Re:Netflix buffering issues: Boxing fans complain about Jake... by mbrumlow on Saturday November 16, 2024 @10:34PM (#42161789)
This.

I can’t tell you the I bet of times things worked because the cache was hot. And a restart or cache invalidation would actually cause an outage.

Caches must be invalidated at a regular interval. Any system that does not do this is heading for some bad days.

Re:Netflix buffering issues: Boxing fans complain about Jake... by lxgr on Saturday November 16, 2024 @05:14PM (#42159844)
It's very hard to do a representative dry run when the most likely potential points of failure are highly load-dependent.

You can try and predict everything that'll happen in production, but if you have nothing to extrapolate from, e.g. because this is your very first large live event, the chances of getting that right are almost zero.

And you can't easily import that knowledge either, because your system might have very different points of failure than the ones external experts might be used to.

Re:Netflix buffering issues: Boxing fans complain about Jake... by leptons on Saturday November 16, 2024 @07:04PM (#42160723)
They could have done a dry run. They could have spun up a million virtual machines somewhere, and tested their video delivery for 30 minutes. Even my small team spins up 10,000 EC2 instances on the regular. Netflix has the money to do much more. I'm sure there are a dozen ways they could have stress-tested this beforehand. It's not like someone sprang this on them last week and they had to scramble to put together a system to do it.
Re:Netflix buffering issues: Boxing fans complain about Jake... by mewpmewp2 on Saturday November 16, 2024 @07:37PM (#42160912)
Maybe they did. We don't know they did not. But problem is that real world traffic will still be always totally different, varied and dynamic in many unexpected ways and a certain link might be under certain effect causing a ripple effect.
Re:Netflix buffering issues: Boxing fans complain about Jake... by lxgr on Saturday November 16, 2024 @08:36PM (#42161245)
How representative is an EC2 instance in a datacenter simulating user behavior really, though?

These would likely have completely different network connectivity and usage patterns, especially if they don't have historical data distributions to draw from because this was their first big live event.

Re:Netflix buffering issues: Boxing fans complain about Jake... by leptons on Saturday November 16, 2024 @11:03PM (#42161881)
>How representative is an EC2 instance in a datacenter simulating user behavior really, though?

Systemic issues causing widespread buffering isn't "user behavior". It's a problem with how Netflix is trying to distribute video. Sure some connections aren't up to the task, and that isn't something Netflix can really control unless they are looking to improve how their player falls-back to lower bitrate video, which could also be tested.

>because this was their first big live event.

That's the point of testing. They should have already had a "big live event" that nobody paid for during automated testing. Instead they seem to have trusted that their very smart and very highly paid developers wouldn't embarrass them based on nothing more than expectations, but they failed. They could have done more rigorous "live" testing before rolling this out to the public.

Re:Netflix buffering issues: Boxing fans complain about Jake... by throwaway2037 on Sunday November 17, 2024 @09:31AM (#42164365)

    > Even my small team spins up 10,000 EC2 instances on the regular.
Woah, this sounds very cool. Can you share more details?
Re:Netflix buffering issues: Boxing fans complain about Jake... by leptons on Sunday November 17, 2024 @03:36PM (#42166996)
I manage ~3000 customized websites based on the same template code. Sometimes we make changes to the template code that could affect the customizations - it is practically impossible to predict what might cause a problem due to the nature of the customizations. We'll take before and after screenshots of every page on every site, so it can get into the 100s of thousands of screenshots. We'll then run a diff on the screenshots to see what changed, reviewing the screenshots with the most significant changes. Then we'll address the problems we find and deploy the fixed release.

When we do these large screenshot operations, the EC2 instances are running for maybe 15 or 20 minutes total. It's not exactly cheap, but losing clients because we broke their site is something we want to avoid. The sites are hosted on a 3rd party service, and we're rate-limited by IP address, so to get this done in a reasonable amount of time we need to spin up 10,000 EC2 instances to distribute the work. We have our own software to manage the EC2 instances. It's honestly pretty simple, but effective.

Re:Netflix buffering issues: Boxing fans complain about Jake... by vel0city on Sunday November 17, 2024 @09:57AM (#42164503)
1) You don't know if they did or did not do this kind of testing. I don't see any proof either way here. You're assuming they didn't.

2) You're assuming whatever issue happened would have been caught by testing on generic EC2 instances in AWS. In the end these streams were going to users on tons of different platforms in lots of different network environments, most of which look nothing like an EC2 instance. Maybe there was something weird with their networking stack on TCL Roku TVs that ended up making network connections reset rapidly chewing up a lot of network resources which led to other issues. What's the EC2 instance type API name for a 55" TCL Roku TV from six years ago on a congested 2.4GHz Wireless N link?

I don't know what happened in their errors. I do know I don't have enough information to say what tests they did or did not run.

Re:Netflix buffering issues: Boxing fans complain about Jake... by yourapostasy on Saturday November 16, 2024 @09:24PM (#42161483)
What I’m seeing in large organizations is tracking dependencies within a team’s scope is better than dependencies between teams, because so many developers between teams are punting on tracking dependencies upon other teams’ artifacts if there isn’t a formal system already in place at the organization to establish contracts between teams along these dependency routes, that automatically handle the state and notifications when changes announcing the intended state change are put through the system. Usually some haphazard dependency representation is embedded into some software and developers call it a day, expecting the software to auto-magically solve a social-technical logistical problem instead of realizing state transitions of the dependencies are not represented and the software could never deliver what they assume.
Re:Netflix buffering issues: Boxing fans complain about Jake... by mbrumlow on Saturday November 16, 2024 @10:31PM (#42161780)
These are the very arguments I get all the time.

1) I want to invalidate caches, I want to know that these systems work. I want to know that my software properly handles this situation.

2) if I have lazy clients. I want to know. And I want to motivate them on updating sooner or figure out how to force update them. I don’t want to not update because some people are slow. I want the norm to be it is updating, so when there is a reason to update, like a zero day, I can have some notion that the updates will work and the lazy clients will not be an issue.

I am not talking about fake or dry runs that go through some portion of motions, I want every aspect of the process to be real.

Performance means nothing if your stuff is down. And any perceived performance gained by not doing proper hygiene is just tweaking the numbers to look better than they really are.

Re:Netflix buffering issues: Boxing fans complain about Jake... by skybrian on Sunday November 17, 2024 @01:43PM (#42165988)
I think it often makes sense to do full releases frequently, but not continuously. For example, Chrome is on an approximately four week schedule, which makes sense for them. Other projects have faster cadences. There is a point of diminishing returns, though, and you seem to be ignoring the downsides.
Re:Netflix buffering issues: Boxing fans complain about Jake... by mbrumlow on Monday November 18, 2024 @11:08PM (#42180037)
I think once a week is good. Maybe once every two weeks.
Re:Netflix buffering issues: Boxing fans complain about Jake... by unoti on Saturday November 16, 2024 @05:30PM (#42159969)
> 1. Always be building: It does not matter if code was not changed...

> 2. Always be releasing...

A good argument for this is security. Whatever libraries/dependencies you have, unpin the versions, and have good unit tests. Security vulnerabilities that are getting fixed upstream must be released. You cannot fix and remove those vulnerabilities unless you are doing regular releases. This in turn also implies having good unit tests, so you can do these builds and releases with a lower probability of releasing something broken. It also implies strong monitoring and metrics, so you can be the first to know when something breaks.

Re:Netflix buffering issues: Boxing fans complain about Jake... by kortilla on Saturday November 16, 2024 @07:59PM (#42161034)
Unless the upstream dependency happens to maintain stable branches, constantly pulling in the latest branches increases your risk of vulnerabilities more than getting the discovered bug patches
Re:Netflix buffering issues: Boxing fans complain about Jake... by caseyohara on Saturday November 16, 2024 @06:21PM (#42160446)
> Whatever libraries/dependencies you have, unpin the versions, and have good unit tests.

Nitpick: unit tests by definition should not be exercising dependencies outside the unit boundary. What you want are solid integration and system tests for that.

Re:Netflix buffering issues: Boxing fans complain about Jake... by 01HNNWZ0MV43FF on Saturday November 16, 2024 @08:17PM (#42161140)
There's two other ways I've seen it phrased:

"Test what you fly, and fly what you test" (Supposedly from aviation)

"There should be one joint, and it should be greased regularly" (Referring to cryptosystems I think, but it's the same principle. Things like TLS will ossify if they aren't exercised. QUIC has provisions to prevent this.)

Re:Netflix buffering issues: Boxing fans complain about Jake... by ravenstine on Saturday November 16, 2024 @09:54PM (#42161623)
There should be a caveat that such this kind of decision should be based on experience and not treated as a rule that juniors might blindly follow. We all know how "fail fast and early" turned out (or whatever the exact phrase was).
Re:Netflix buffering issues: Boxing fans complain about Jake... by _dark_matter_ on Saturday November 16, 2024 @02:23PM (#42158542)
Agreed. This is a management failure, full stop. Unbelievable that they'd expect engineering to handle a single Livestream event of this magnitude.
Re:Netflix buffering issues: Boxing fans complain about Jake... by ignoramous on Saturday November 16, 2024 @03:16PM (#42158930)
> ...the tech team has no experience in

Unless Netflix eng decides to release a public postmorterm, we can only speculate. In my time organizing small-time live streams, we always had up to 3 parallel "backup" streams (Vimeo, Cloudflare, Livestream). At Netflix's scale, I doubt they could simply summon any of these providers in, but I guess Akamai / Cloudflare would have been up for it.

Re:Netflix buffering issues: Boxing fans complain about Jake... by geor9e on Saturday November 16, 2024 @03:07PM (#42158854)
They've been doing live events since 2023. But it's hard to be prepared for something that's never been done by anyone before — a superbowl scale event, entirely viewed over the internet. The superbowl gets to offload to cable and over the air. Interestingly, I didn't have any problems with my stream. So it sounds like the bandwidth problems might be localized, perhaps by data center or ISP.
Re:Netflix buffering issues: Boxing fans complain about Jake... by burntalmonds on Saturday November 16, 2024 @04:36PM (#42159567)
Yeah, I think people are incorrectly assuming that everyone had the same experience with the stream. I watched the whole thing and only had a few instances of buffering and quality degradation. Not more than 30 seconds total during the stream.
Re:Netflix buffering issues: Boxing fans complain about Jake... by DharmaPolice on Saturday November 16, 2024 @05:04PM (#42159760)
Even if it was only 30% of people had a problem that's still millions of unhappy users. Not great for a time sensitive event.

Also, from lurking in various threads on the topic Netflix's in app messages added to people's irritation by suggesting that they check their WiFi/internet was working. Presumably that's the default error message but perhaps that could have been adjusted in advance somehow.

Re:Netflix buffering issues: Boxing fans complain about Jake... by positr0n on Saturday November 16, 2024 @09:43PM (#42161566)
One of the times I reloaded the page I got a raw envoy error message!
Re:Netflix buffering issues: Boxing fans complain about Jake... by ta1243 on Sunday November 17, 2024 @07:17AM (#42163798)
The point is that if the problem was different depending on the user, it will be in a distribution layer, not in the encoding or production layer

That eliminates a whole raft of problems.

Re:Netflix buffering issues: Boxing fans complain about Jake... by mastazi on Saturday November 16, 2024 @05:10PM (#42159816)
Maybe they considered this event as a rehearsal for the upcoming NFL streams which I am guessing might have a wider audience
Re:Netflix buffering issues: Boxing fans complain about Jake... by kpierce on Wednesday November 20, 2024 @10:10PM (#42200724)
Yes I agree that fight had a great deal of interest but the nfl is their real goal.
Re:Netflix buffering issues: Boxing fans complain about Jake... by firesteelrain on Saturday November 16, 2024 @05:58PM (#42160225)
I had issues here and there but there was workarounds. Then, towards the end, the quality either auto negotiated or was forced down to accommodate the massive pull.
Re:Netflix buffering issues: Boxing fans complain about Jake... by elcritch on Saturday November 16, 2024 @09:15PM (#42161436)
I suspect a lot of it could be related to ISP bandwidth. I streamed it on my phone without issue. Another friend put their TV on their phone’s WiFi which also worked. Could be partly that phone hotspots lower video bandwidth by default.

I suspect it’s a bit of both Netflix issues and ISPs over subscribing bandwidth.

Re:Netflix buffering issues: Boxing fans complain about Jake... by positr0n on Saturday November 16, 2024 @09:42PM (#42161557)
I would guess the majority of the streamed bandwidth was sourced from boxes like these in ISP's points of presences around the globe: https://openconnect.netflix.com/en/

So I agree the problems could have been localized to unique (region, ISP) combinations.

Re:Netflix buffering issues: Boxing fans complain about Jake... by uep on Sunday November 17, 2024 @10:40AM (#42164734)
My suspicion is the same as yours, that this may have been caused by local ISPs being overwhelmed, but it could be a million other things too. I had network issues. I live in a heavily populated suburban area. I have family who live 1000+ miles away in a slightly less populated suburban area, they had no issues at all.
Re:Netflix buffering issues: Boxing fans complain about Jake... by patrick451 on Sunday November 17, 2024 @12:31PM (#42165472)
The ISP hypothesis doesn't make sense to me. I could not stream the live event from Netflix. But I could watch any other show on netflix or youtube or hulu at the same time.
Re:Netflix buffering issues: Boxing fans complain about Jake... by geor9e on Sunday November 17, 2024 @06:11PM (#42168193)
Some ISPs have on-site Netflix Open Connect racks. The advantage of this is that they get a high-priority quality of service data stream into the rack, which then serves the cached content to the ISP customers. If your ISP doesn't have a big enough Netflix rack and it gets saturated, then you're getting your streams at the whim of congestion on the open internet. A live stream is a few seconds of video downloaded, and it has to make it over the congestion of the internet in a few seconds and then repeat. If a single one of these repeats hits congestion and gets delayed, you see the buffering spinning wheel. Other shows, on the other hand, can show the cached Netflix splash animation for 10 seconds while they request 20 minutes of cache until they get it. So, dropped packets don't matter much. Even if the internet is seeing congestion every couple of minutes, delaying your packets, it won't matter as non-live content is very flexible and patient about when it receives the next 20-minute chunk. I'm not an ISP or Netflix engineer, so don't take these as exact numbers. I'm just explaining how the "bandwidth problems might be localized" hypothesis can make sense from my general understanding.
Re:Netflix buffering issues: Boxing fans complain about Jake... by MisterBastahrd on Saturday November 16, 2024 @02:57PM (#42158782)
The WWE is moving their programming to Netflix next year. If I were them, I'd be horrified at what I saw.
Re:Netflix buffering issues: Boxing fans complain about Jake... by giantg2 on Saturday November 16, 2024 @05:27PM (#42159942)
Wow, building talent from within? I thought that went out of fashion. I think companies are too impatient to develop their employees.
Re:Netflix buffering issues: Boxing fans complain about Jake... by eh9 on Saturday November 16, 2024 @06:18PM (#42160430)
that’s difficult to reproduce at scale; there are only so many “super bowl” events in a calendar year
Re:Netflix buffering issues: Boxing fans complain about Jake... by don-code on Saturday November 16, 2024 @07:47PM (#42160978)
Sometimes this just isn't feasible for cost reasons.

A company I used to work for ran a few Super Bowl ads. The level of traffic you get during a Super Bowl ad is immense, and it all comes at you in 30 seconds, before going back to a steady-state value just as quickly. The scale pattern is like nothing else I've ever seen.

Super Bowl ads famously seven million dollars. These are things we simply can't repeat year over year, even if we believed it'd generate the same bump in recognition each time.

Re:Netflix buffering issues: Boxing fans complain about Jake... by NoPicklez on Sunday November 17, 2024 @06:58PM (#42168444)
I think Netflix have a fair bit of organisational muscle, perhaps the fight was considered not as large of an event as the NFL streams would be in the future.

Also, "No experience in" really? You have no idea if that's really the case