relix42

It was done a tested more than once. As I recall, it took quite a bit to get Linux to perform to the level that BSD was performing for (a) this use can and (b) the years of investment Netflix had already put into the BSD systems.

So, could Linux be tweaked and made as performant for _this_ use case. I expect so. The question to be answered is _why_.

Chaos Monkey was explained to me as "The (virtual) monkey swinging around the (virtual) data center unplugging the (virtual) cables". It very well could have been based on Robbins. The point wasn't originality - the point was to have the same benefit in our burgeoning cloud environment.

Initially we wrote the library to help us answer the question about what change may be causing impact to our customers' experiences. Out of the billions of time series metrics we have we knew there were about 10,000 or so likely (for this initial use case), possible candidates and we wanted to reduce that set as far as possible and either (1) find /the/ candidate for problem or (2) produce a short list of things for the humans to look into. In order to get to the point where we could ensure high likelihood of apropos correlations, we needed to do some work on the signals first.

First we detect if any of the possible candidate signals were born or died in the interesting time period. We use these time points to reduce the window we'll use to pass to the correlation functions. We can also detect any changepoints in the time series and apply similar logic. Once we've determined the best window bounds for each candidate signal, we use pearson and spearman correlation functions to get a score for the pair of signals -- the initial signal that started the inquiry and the candidate signal using the determined time window.

The code is about 98% data preparation, signal analysis, and window determination and about 2% correlation work.

(I've tried to summarize quite a bit, let me know if you'd like clarifications or have other questions.)

In my opinion, it doesn't count unless its in production.

Why? Your customers use your production environment, not your test environment. Something will cause loss of an instance for you: * Mistaken termination * AWS retirement (and you missed the email) * Cable trip in the data center * <Something else we can come up with> * <This list goes on>

So, vaccinate against the loss of an instance cratering your service. Give your prod environment a booster shot (with Chaos Monkey or something like it) every hour of every day. Then, when anything from the above list happens you're infrastructure handles it gracefully and without intervention. Continued booster shots ensure that this stability continues through config changes, software version changes, OS changes, tooling changes, etc.

I think the better question is "Why wouldn't you do this?"

Not deployment or spinnaker related

To put a finer point on the OpenConnect comments - Netflix doesn't stream any video from AWS. All video bits are served by the OpenConnect CDN.

This has more to do with licensing restrictions than with technology.

When I had 5 weeks ago I never took all of it. The environment didn't support it. It was very much the "here are 5 weeks of vacation we hope you won't use".

Now I can a week or two here and there, a few days off, a long weekend, etc. It's my job to plan it out, balance my work, and go. Thinking back on last year (without going over my calendar) I'd estimate I took between 5 and 6 weeks of total time off.

I can speak for me and others I know rather well - I like what I do. I enjoy going to work, finding things to fix, and fixing them. There are those things that I don't enjoy doing, but, they're not the majority by a long shot. So, six months away wouldn't be that attractive.

Along with the that, one of the tenets of the business is "Freedom and Responsibility". If you can be responsible taking six months off then you are free to do so. I know people that take 4 weeks off at a time without issue. They plan their projects and commitments, communicate to their team, and make sure they're ready to be out for 4 weeks. Then, they leave for 4 week.

My management chain encourages me to take time away from the office. Vacations, conferences, et al. I determine my workload, so, if the day-to-day is stressful, I need to change something.

I've seen quite a few reactions to the Netflix unlimited PTO policy that are similar to the reactions to the new Maternity and Paternity policy. Many of these opinions are the likes of:

* "I'm suspicious of this."

* "No one takes any vacation at Netflix because of this."

* "Unlimited == None"

* "Peer pressure means that everyone works 52 weeks a year."

* "Without a PTO policy you can't cash out unused vacation. This is a rip off."

The thing I haven't seen in these statements are "I work at Netflix and this is what it is really like"...

Here we go - I work at Netflix (as an engineer) and this is what is really like:

* I'm pressured to take vacations.

* Managers are taught that they are examples to the teams therefore they must take regular vacations.

* I take more vacation now than I did when I had 2/3/4/5 weeks a year of stated, paid, vacation.

Questions I've answered about this: * Does my team fall apart when someone leaves for 4 weeks? No. They're adults and they know how to prepare to be away for a while.

* People must raise their eyebrows at you when you leave! No.

* You get called all the time right so you keep your laptop with you on vacation? No. They're adults and they know how to get along without me for a while.

* You must do tons of extra work when people go on vacation! No. People don't just drop stuff and run. They prepare, get stuff ready, postpone things until they're back, etc.

* Managers must "encourage" you not to take vacations. Nope, opposite.

* You feel irresponsible taking time off then. No I don't.

* People must leave for months a time right? The policy gets really abused! No, stop it. Assuming the extreme case must be the common case is silly. You're smarter than that.

* This can't possibly be true. You're a liar/shill/idiot! Next.

* This doesn't/can't work at my company. Therefore, it can't work at yours! Netflix corporate culture is likely very different than your company. Take a look at the culture deck presentation.

This type of policy likely can't work everywhere.

It may not work everywhere in the valley.

It does work here.

[edit] formatting fail

Both

If you're good and what you do and have passed the interview process, you'll be fine. The fact that you are self-aware enough to understand the high performance culture and what it takes to meet that bar further indicates that you'll fit in.

There isn't a stack rank system at all. Nothing like it in fact.

High turn over - more so in the customer service side of the business; not the engineering side.

Source: work at Netflix, interview a lot of engineering candidates, have thrown things at diab0lic.

[edit 1] Answering a few other questions:

Unlimited vacation - yup it's true and encouraged. Being unable to cash out 2-3 weeks of vacation isn't something I see as a big deal. I take more time off than that. My co-workers not my managers make me feel bad about it. Why? They're on vacation.

Work and work load - I choose much of what I work on. My job is to make things better at Netflix and I'm trusted to do so.

High salary - many other companies put some of your remuneration in RSUs, options, or other vehicles that may create future benefit. So you're gambling and you have to stick around to see if it pays off. Netflix pays you your salary in cash right now; no golden handcuffs in the form of vesting schedules or stock levels. (There is an optional options program)