I think that employees actually makes less mistakes and are more productive if they don't have be worried about being fired for making a mistake.
> A young executive had made some bad decisions that cost the company several million dollars. He was summoned to Watson’s office, fully expecting to be dismissed. As he entered the office, the young executive said, “I suppose after that set of mistakes you will want to fire me.” Watson was said to have replied,
> “Not at all, young man, we have just spent a couple of million dollars educating you.” [1]
All depends on how leadership views employee growth
[1] http://the-happy-manager.com/articles/characteristic-of-lead...
Not a great argument because many people view one of the primary responsibilities of local government is to maintain the roads (in USA). If they cannot properly budget and allocate money, regardless if the tax increase worked, it was the wrong way to fix the problem. With this mindset government can fix every problem by raising taxes.... Not acceptable to most people.
If people really think that the government can maintain roads with no money, assuming they don't have that money, I don't know what to say.
If their primary purpose is to take care of the roads, that should be one of the first items that gets funded with taxes they already collect, therein lies the problem people have. It's not like they have no money, it was improperly allocated to the point where they were in the negative to meet the needs required of them. We are not a country with a lot of social services, we have very few. It a case of the government not doing their jobs well and taking more money cover that fact up.
I am not familiar with this particular instance. But the story about Clinton as it stands is not really relevant. Much like this sub-thread.
Do you really think they brought in "no" money? That's ridiculous. The government should figure out how to waste less of the existing taxes before demanding more.
Im not sure who caused the budget deficit in the first place, but he is the one that took more money from citizens to fix a problem that should have been fixed by reallocating existing funds.
That said, governments do spend money that doesn't exist as a matter of routine. That's why the Fed exists.
I did fire an employee who deleted the entire CVS repository.
Actually, as you say, I didn't fire him for deleting the repo. I fired him the second time he deleted the entire repo.
However there's a silver lining: this is what led us (actually Ian Taylor IIRC) to write the CVS remote protocol (client / server source control). before that it was all over NFS, though the perp in question had actually logged into the machine and done rm -rf on it directly(!).
(Nowadays we have better approaches than CVS but this was the mid 90s)
The first time it happened, we didn't understand what, exactly, had caused it. The database directory was just gone, and it seemed to have gone around 11pm. I (not they!) discovered this and we scrambled to recover the data. We had replication, but for some reason the guy on call wasn't able to restore from them -- he was standing in for our regular ops guy, who was away on site with another customer -- so after he'd struggled for a while, I said screw it, let's just restore the last dump, which fortunately had run an hour earlier; after some time we were able to get a new master set up, though we had lost one out of data. Everyone went to bed around 1am and things were fine, the users were forgiving, and it seemed like a one-time accident. They promised that setting up a new replication slave would happen the next day.
Then, the next day, at exactly 11pm, the exact same thing happened. This obviously pointed to a regular maintenance job as being the culprit. It turns out the script they used to rotate database backup files did an "rm -rf" of the database directory by accident! Again we scrambled to fix. This time the dump was 4 hours old, and there was no slave we could promote to master. We restored the last dump, and I spent the night writing and running a tool that reconstructed the most important data from our logs (fortunately we logged a great deal, including the content of things users were creating). I was able to go bed around 5am. The following afternoon, our main guy was called back to help fix things and set up replication. He had to travel back to the customer, and the last things he told the other guy was: "Remember to disable the cron job".
Then at 10pm... well, take a guess. Kaboom, no database. Turns out they were using Puppet for configuration management, and when the on-call guy had fixed the cron job, he hadn't edited Puppet; he'd edited the crontab on the machine manually. So Puppet ran 15 mins later and put the destructive cron job back in. This time we called everyone, including the CEO. The department head cut his vacation short and worked until 4am restoring the master from the replication logs.
We then fired the company (which filed for bankruptcy not too long after), got a ton of money back (we threatened to sue for damages), and took over the ops side of things ourselves. Haven't lost a database since.
What happened was that one morning when people were logging in to the DECstation they noticed that things didn't quite work. Pretty much everything they normally did (like running Emacs, compiling things, etc) worked, but other, slightly more rare things just didn't work. The binaries seemed to be missing. It was all very strange.
We spent some time looking into it and finally we figured out what had happened. During some mantenance, the root directory of the DECstation had been NFS-mounted to the VAX, and the mount point was under /tmp. I don't remember who did it, but it's not unlikely that it was me. During the night, the /tmp cleanup script had run on the VAX which deleted all files that had an atime (last access time) of more than 5 days. This meant that all files the DECstation needed to run, and all the files that were used during normal operation were still there, but anything slightly less common had been deleted.
This obviously taught me some lessons, such as never mount anything under /tmp, never NFS mount the root directory of anything and never NFS mount anything with root write permissions. The most important thing about sysadmin disasters are that you learn something from them.
$ rm -rf / home/myusername/mylargedir/
(note the extra space after slash)The real solution is comprised of:
* backups (which are restored periodically to ensure they contain everything)
* proper process which makes accidental removal harder (DCVS & co.)What most people don't realize is that very few places have a real (tested) backup system.
_goes off to check backups_
And then commit it.
It's a good habit.
1) Write a select statement capturing the rows you want to modify and verify them by eyeball
2) (Optional) Modify that statement to select the unchanged rows into a temp table to be deleted in a few days
3) Wrap the statement from step 1 in a transaction
4) Modify the statement into the update or delete
5) Check that rowcounts haven't changed from step 1
6) Copy-and-paste the final statement into your ticketing or dev tracking system
7) Run the final statement
It may be overkill, but the amount of grief it can save is immeasurableI do what your coworker did and it's a great feeling when you get the "451789356 rows updated" message inside a transaction where you are trying to change Stacy's last name after her wedding and all you have to do is run a ROLLBACK.
Then it's time to go get a coffee and thank your deity of choice.
- With shells, I prefix risky commands on production machines with #, especially when cutting and pasting
- Same for committing stuff into VCS, especially when I'm cherrypicking diffs to commit
- Before running find with -delete, run with -print first. Many other utilities have dry-run modes
Then open a transaction around the update with that same where clause, check the total number of rows updated matches the earlier select, then commit.
This approach definitely reduces your level of anxiety when working on a critical database.
UPDATE ImportantTable SET
ImportantColumn = ImportantColumn
WHERE Condition = True
Check the rows affected, then change it to: UPDATE ImportantTable SET
ImportantColumn = NewValue
WHERE Condition = TrueYears ago we had an intern for a time, and he set up our backup scripts on production servers. He left after a time, we deleted his user, and went on our merry way. Months later, we discover the backups had been running under his user account, so they hadn't been running at all since he left. A moment of "too busy" led to a week of very, very busy.
We managed to restore all but about a dozen users from backup, and sent a sheepish email to the rest asking them to reset their passwords.
That was the day I started backing up everything.
alias rm='echo "This is not the command you are looking for."; false'
I install trash-cli and use that instead.Of course this does not prevent other kinds of accidents, like calling dd to write on top of the /home partition... ok, I am a mess :)
* making "--preserve-root" the default... :-)
> cd /mnt/backup
> sudo rsync -a --delete user@remote:some/$dir/ $dir/
Only to see the local machine become pretty much empty when $dir was not set.
Funny to still see Apache etc still running in memory despite any related files missing.
For next time Apache is hosting a phantom root dir. ;) These things happen to all of us. We just have to be prepared.
With NFS Version 3, aka NeFS, instead using rlogin to rm -rf on the server, the perp could have sent a PostScript program to the server that runs in the kernel, to rapidly and efficiently delete the entire CVS tree without requiring any network traffic or even any context switches! ;)
http://www.donhopkins.com/home/nfs3_0.pdf
The Network Extensible File System protocol(NeFS) provides transparent remote access to shared file systems over networks. The NeFS protocol is designed to be machine, operating system, network architecture, and transport protocol independent. This document is the draft specification for the protocol. It will remain in draft form during a period of public review. Italicized comments in the document are intended to present the rationale behind elements of the design and to raise questions where there are doubts. Comments and suggestions on this draft specification are most welcome.
The Network File System The Network File System (NFS™* ) has become a de facto standard distributed file system. Since it was first made generally available in 1985 it has been licensed by more than 120 companies. If the NFS protocol has been so successful why does there need to be NeFS ? Because the NFS protocol has deficiencies and limitations that become more apparent and troublesome as it grows older.
1. Size limitations.
The NFS version 2 protocol limits filehandles to 32 bytes, file sizes to the magnitude of a signed 32 bit integer, timestamp accuracy to 1 second. These and other limits need to be extended to cope with current and future demands.
2. Non-idempotent procedures.
A significant number of the NFS procedures are not idempotent. In certain circumstances these procedures can fail unexpectedly if retried by the client. It is not always clear how the client should recover from such a failure.
3. Unix®† bias.
The NFS protocol was designed and first implemented in a Unix environment. This bias is reflected in the protocol: there is no support for record-oriented files, file versions or non-Unix file attributes. This bias must be removed if NFS is to be truly machine and operating system independent.
4. No access procedure.
Numerous security problems and program anomalies are attributable to the fact that clients have no facility to ask a server whether they have permission to carry out certain operations.
5. No facility to support atomic filesystem operations.
For instance the POSIX O_EXCL flag makes a requirement for exclusive file creation. This cannot be guaranteed to work via the NFS protocol without the support of an auxiliary locking service. Similarly there is no way for a client to guarantee that data written to a file is appended to the current end of the file.
6. Performance.
The NFS version 2 protocol provides a fixed set of operations between client and server. While a degree of client caching can significantly reduce the amount of client-server interaction, a level of interaction is required just to maintain cache consistency and there yet remain many examples of high client-server interaction that cannot be reduced by caching. The problem becomes more acute when a client’s set of filesystem operations does not map cleanly into the set of NFS procedures.
1.2 The Network Extensible File System
NeFS addresses the problems just described. Although a draft specification for a revised version of the NFS protocol has addressed many of the deficiencies of NFS version 2, it has not made non-Unix implementations easier, not does it provide opportunities for performance improvements. Indeed, the extra complexity introduced by modifications to the NFS protocol makes all implementations more difficult. A revised NFS protocol does not appear to be an attractive alternative to the existing protocol.
Although it has features in common with NFS, NeFS is a radical departure from NFS. The NFS protocol is built according to a Remote Procedure Call model (RPC) where filesystem operations are mapped across the network as remote procedure calls. The NeFS protocol abandons this model in favor of an interpretive model in which the filesystem operations become operators in an interpreted language. Clients send their requests to the server as programs to be interpreted. Execution of the request by the server’s interpreter results in the filesystem operations being invoked and results returned to the client. Using the interpretive model, filesystem operations can be defined more simply. Clients can build arbitrarily complex requests from these simple operations.
- Employee was error prone and this mistake was just the biggest one to make headlines. Could be from incompetence or apathy.
- Impacted clients demanded the employee at-fault be terminated.
- Deterrence: fire one guy, everyone else knows to take that issue seriously. Doesn't Google do this? If you leak something to press, you're fired, then a company email goes out "Hey we canned dude for running his mouth..."
It's better to engage the known and perhaps questionable justifications than to "never understand".
Case 2: no competent manager would fire an employee who made a mistake to satisfy clients. They may move the employee to a role away from that client, but it would be insanity to allow the most unreasonable clients to dictate who gets fired. Any manager who does what you suggest should expect to have lost all credibility in the eyes of their team.
Case 3a: A leak to the press is a purposeful action. Firing for cause is perfectly reasonable. Making a mistake is not a purposeful action.
Case 3b: If you want to convey that a particular type of mistake is serious, don't do so by firing people. Do so with investments in education, process, and other tools that reduce the risk of the mistake occurring, and the harm when the mistake occurs. Firing somebody will backfire badly, as many of your best employees will self-select away from your most important projects, and away from your company, as they won't want to be in a situation where years of excellent performance can be erased with a single error.
Case 3a: Good distinction, a conscious leak is not a mistake. It's possible for a leak to be accidental though, say under alcohol, lost laptop, or just caught off guard by a clever inquisitor.
Case 3b: Firing has the effects you mention, but it also has the effect of assigning gravity to that error. I'm not claiming the benefits outweigh the drawbacks, but some managers do.
I'm not a proponent of the above, but it's good to understand the possible rationale behind these decisions.
If you're going to have firing offenses, spell those out. E.g. breaking the law, violating some set of rules in the handbook, whatever., so that people can at least know there's a process or sensibility to the actions.
If people can be fired for making a mistake, and that wasn't laid out at the outset, then they're just not gonna trust the stability of your workplace.
This is going to depend on the severity, cost, budget, importance of the role filled, etc., but I think it's probably one of the only semi-plausible justifications for firing based on things that do not reflect a serious and ongoing competency or legal issue.
A mistake is made, and a material loss has been incurred. This sucks. Been there, done that, didn't get the t-shirt because we couldn't afford such a luxury. I watched my annual bonus evaporate because of somebody else's cock-up.
But there's no reason to believe that firing the mistake-maker is the best move. Maybe the right move is to find money somewhere else (cutting a bunch of discretionary, pushing some expenses into the future, reducing some investment), or maybe it's to ask a few people to take pay cuts in return for some deferred comp. Or maybe it's to lay-off somebody who didn't do anything wrong, but who provides less marginal value to the company.
But it'd be one hell of a coincidence if, after an honest to god mistake, the best next action was to fire the person who made the mistake. After all, if they were in a position to screw your company that hard, they almost certainly had a history of being talented, highly valued, and trustworthy. If they weren't good, you wouldn't have put them in a place where failure is so devastating.
Yeah, I'm not saying it necessarily or even probably is. I'm saying that reality sometimes makes it so that we have to make these compromises.
What was the fired person doing? Presumably they were performing required work otherwise the company wouldn't have been paying them in the first place.
That means you know need to pay to replace which costs more than keeping an existing employee. Or you could divide their responsibilities among the remaining employees but if you thought you could do that you would have already laid them off without waiting for them to mess something up.
If I see someone getting axed for making a mistake, I'd be making a mistake if I didn't immediately start firing up the resume machine.
I've never heard of this happening. I've heard of people fired for taking photographs (or stealing prototypes!) of confidential products and handing them to journalists.
"Sir beg my pardon for asking but why did you give Smith 50 press-ups for asking a question? You said that there were no stupid questions. Sir."
"I gave him the press-ups the SECOND time he asked. Will you need to ask again?"
The IRL takeaway is that if you don't open your mouth, you don't get punished. If you do open your mouth, you might get punished.
"What kind of Mickey Mouse show is Manager Mark running over there?" asks colleague Claire, "Isn't that guy Ryan the same one that screwed up the TPS reports last year?"
On the other hand, if Mark fires Ryan, then mark is a decisive manager. Even if the total number of major errors is higher, then still there will not be a risk of letting being known as a manager that let's people screw up multiple times.
Furthermore, a lot of individual error's are seen in an institutionalised "systems" framework - given that people invariably will make mistakes, how can we set up the environment/institutions/systems so that errors are not catastrophic.
Not sure how that applies to movie animation, to be honest, but not primarily looking for whom to blame was certainly a very good move.
Of course, if the problem is just a borderline behavior of the pilot or co-pilot, it'll be fantastic if we can get him off the circuit before he locks the captain outside and programs the plane to crash against a mountain. Or not to stretch fuel limits so that he will fall out of gas.
But... if we can also learn how to make a out-of-gas plane land and survive, and the cost is "let's not put this pilot into jail, because it's better to learn how to save more lives", I prefer this approach. Probably you'll be able to get the pilot from some other behavior.
Yes, it does. Many aviation crashes are attributed to "pilot error". There's only so much you can do with procedures and such; at some point, the pilot has to be held accountable for screwing up, and investigations do exactly that many times.
Usually, in major events, you're looking at commercial airliners with a pilot and co-pilot and in those cases, it's usually something much worse than a mistake by the pilot, and frequently several bad things happening at once. But in general aviation, where you have one pilot, frequently non-commercial, flying a small aircraft, the cause is frequently just "pilot error". A common example of this is the pilot running out of fuel because they did their calculations wrong. It happens frequently with private pilots, and in a Cessna you can't just pull over when you run out of gas.
For example, the first fatal 747 crash, the Lufthansa coming down upon departure from Nairobi, happened almost certainly because the flight crew did not extend the leading edge flaps. Clear case of pilot error. If that response had been it, it would have happened again (in fact, it did happen at least twice before, but at lower altitude airports where the aircraft performance was enough for the crew to depart without accident).
Instead, it was acknowledged that the whole system could be improved, and Boeing put in a take-off configuration warning.
Similarly, AF 347 over the Atlantic - sure, you can argue that the pilot in the right seat should not have pushed the stick forward, and that it was entirely his fault, case closed. But maybe one can, instead, improve the whole system, the HCI, etc.
Edit: typo
Not in general aviation. You're not going to get all the Cessna 172 owners to modify some part of their plane to make it better and avoid some incident where some yahoo private pilot did something dumb and crashed. It's hard enough just getting privately-owned aircraft to be properly serviced. Many of them are many decades old and quite primitive. You're not going to improve "the whole system", the HCI, etc. in some airplane made in 1940 or whenever.
Finally, attributing an incident to "pilot error" doesn't automatically mean that there weren't contributing factors or that things couldn't be done better.
I can't shake the suspicion that the Airbus man-machine interface and programming is partly to blame - possibly only when pushed into an extreme configuration, and certainly just as a topping on other factors.
It's clear however, that it's politically and economically impossible to ground all machines made by the European union's prestige project, Airbus.
Remember the AirAsia from Surabaya, where they got themselves into an upset because of a tiny crack in the rudder-travel limiter unit ("topping on other factors"), then apparently made basically the same mistake as AF 447, one pilot pulling full back all the way down.
> The NTSB does not assign fault or blame for an accident or incident; rather, as specified by NTSB regulation, “accident/incident investigations are fact-finding proceedings with no formal issues and no adverse parties ... and are not conducted for the purpose of determining the rights or liabilities of any person.” 49 C.F.R. § 831.4.
(This is on first (non-title) page of any NTSB accident report. [1])
Of course, the NTSB determines "Probably Cause", and makes safety recommendations. But the point is not to blame the pilot. Notice also that enforcement action is taken by a different agency, the FAA, not the NTSB.
Lastly, the whole ASR reporting system is set up to maximise the information gathered and minimise future accidents, while giving some dispensation to pilots that have made mistakes.
[1] see e.g. e.g. https://www.ntsb.gov/investigations/AccidentReports/Reports/...
Edit: add > to indicate quote.
Cause: cargo was not secured enough and slid back during take-off, shifting centre of gravity, stall, crash. Blame: loadmaster. Done. Or are we?
No: Loadmasters are not FAA-certificated (a gap in the system). The operator procedures were inadequate. FAA oversight over these cargo operations was deficient, one reason being that the FAA inspectors were insufficiently trained. So, suddenly "blame" rests not only with the loadmaster (who perished in the crash, btw), but with the system, the operator, FAA procedures, FAA training, etc.
That's precisely the idea: if a pilot makes errors so grave as to endanger the aircraft, how come the airline training/monitoring in place did not pick up indications earlier?
The people that helped overcome the "pilot error, case closed"-mindset must be thanked (among others) for making aviation today as safe as it is.
Assigning "Cause" is an important part of accident analysis. There have been many cases of a pilot with proper aeronautical decision making processes, and due care and caution, still making a piloting error resulting in a mishap.
Classic example would be TACA 110 [0] Weather related factors, and a flaw in the engine design caused both engines to fail. Rushing the restart procedure resulting in a "hung start" of both engines and subsequent overheat. Thanks to the skillful flying of Capt. Carlos Dardano, and his crew, this 737 made one of the most successful dead-stick landings in history. Capt. Dardano was not to "blame" for the mishap, but he did make errors that were contributing factors.
The 737 needed an engine replaced, but was able to be flown out and fully repaired within weeks of the mishap. Southwest Airlines retired the aircraft in December 2016, with over 27 years of uneventful service.
Same goes for every tech company I have worked at. I have never been in a post-mortem meeting where the goal was to allocate blame. It was always emphasized that the goal of the meeting was to improve our process to make sure it never happens again, not punish the party responsible.
Excerpts from Wiki:
> In March 2008, Bernard Farret, a deputy prosecutor in Pontoise, outside Paris, asked judges to bring manslaughter charges against Continental Airlines and two of its employees – John Taylor, the mechanic who replaced the wear strip on the DC-10, and his manager Stanley Ford – alleging negligence in the way the repair was carried out.
> At the same time charges were laid against Henri Perrier, head of the Concorde program at Aérospatiale, Jacques Hérubel, Concorde's chief engineer, and Claude Frantzen, head of DGAC, the French airline regulator. It was alleged that Perrier, Hérubel and Frantzen knew that the plane's fuel tanks could be susceptible to damage from foreign objects, but nonetheless allowed it to fly.
> Continental Airlines was found criminally responsible for the disaster by a Parisian court and was fined €200,000 ($271,628) and ordered to pay Air France €1 million. Taylor was given a 15-month suspended sentence, while Ford, Perrier, Hérubel and Frantzen were cleared of all charges. The court ruled that the crash resulted from a piece of metal from a Continental jet that was left on the runway; the object punctured a tyre on the Concorde and then ruptured a fuel tank. The convictions were overturned by a French appeals court in November 2012, thereby clearing Continental and Taylor of criminal responsibility.
> The Parisian court also ruled that Continental would have to pay 70% of any compensation claims. As Air France has paid out €100 million to the families of the victims, Continental could be made to pay its share of that compensation payout. The French appeals court, while overturning the criminal rulings by the Parisian court, affirmed the civil ruling and left Continental liable for the compensation claims.
I've found that that helps morale, as there's a sense of shared responsibility, but there's no blaming people for problems where I work, so I haven't actually seen what happens when people are searching for the culprit.
The usual process is "this happened because of this, this and this all went wrong to cause us to not notice the problem, and we've added this to make it less likely that it will happen again". If you have smart, experienced people and you get problems, it's because of your processes, not the people, so the former is what you should fix.
One day at $megacorp a bug caused a production outage and a project manager sent an email to the team calling out the engineer who committed the change and asking us all to be more careful in the future. That manager was immediately reprimanded both in the email chain and in private.
I find the culture of shared responsibility to be one of the best qualities of our industry, even if it isn't universal.
I left and found out two months later from a friend he had managed to take down almost every single server in the place for which he had access. Even the legacy don't touch systems that just boot and run equipment.
Be pretty.
Since Catmull has an engineering background (his PhD involved the invention of the Z-buffer, and he was doing computer graphics before anyone knew anything about it), he understands that mistakes and failed projects, when combined with an forthright and collaborative feedback loop, are not problematic detours, but rather necessary mile markers on the path to real innovation. We'd be so much further ahead if we put more men like Catmull in charge of things.
The biggest problem with reading Creativity Inc. is that it will rekindle the hope that there may be a sane workplace out there somewhere, when practically speaking, we know that few of us will ever find employment in one. It gave me a number of disquieting feelings as I read that the attributes of a workplace that all good engineers crave actually can and sometimes do exist out there. I had convinced myself that these things were myths, so now I'm sad that my boss isn't Ed Catmull.
That said, I do believe some evaluation and/or discipline would've been appropriate in this case, not for the person who accidentally executed a command in the wrong directory, but for the people who were supposed to be maintaining backups and data integrity.
Assuming that your primary job duties involve data integrity and system uptime, having non-functional backups of truly critical data stretches beyond the scope of "mistakes" and into the scope of incompetence.
It is, I'm sure, very possible that no one was really assigned this task at Pixar and that it would therefore by improper to punish anyone in particular for the failure to execute it, but I do believe there is a limit between mistakes en route to innovation and negligence. My experience has been that most companies strongly take one tack or the other: they either let heads roll for minor infractions (and thus never allow good people to get established and comfortable), or they never fire anyone and let the dead weight and political fiefdoms gum up the works until the gears stop altogether. There needs to be a balance, and that's a very hard thing to find out there.
If indeed there was no-one assigned this task, then it was a mistake of negligence on the part of Pixar's management at the time. I'm not saying that to be snippy — that is exactly the job of management: to build the systems and processes required for employees to achieve the firm's goals. Proper backup and restore of data is one of those processes.
An executive usually requires a "Come to Jesus" moment like this one, where the entire company teeters on the precipice due to lax backup or security policies, to really have the importance impressed upon him or her. At that point, they are generally much more supportive, though sadly, this too can start to fade if the sysadmins do their job too well.
I don't want anyone to come away thinking that most companies have solved these problems. It's definitely not the case, even in large, established companies. Security and backups continue to get little attention until it's too late.
We really need to call a celebrity in MBA circles and get that person to run a seminar meant to scare the pants off the execs.
Wasn't Catmull involved in wage-fixing? [http://www.cartoonbrew.com/artist-rights/ed-catmull-on-wage-...]
"“Like somehow we’re hurting some employees? We’re not,” Catmull said. “While I have responsibility for the payroll, I have responsibility for the long term also,” Catmull said. “I don’t apologize for this. This was bad stuff.”"
I can't find the other interview, but in a later interview he makes it clear his job is to worry about Disney's profitability.
I worked in visual effects for five years and loved every moment. People who choose that profession are fun, creative, passionate artistic, super energetic, crazy, smart, pull off the nearly impossible every project, out of the ordinary in every way, and crazy (listing that one twice). I miss the people. When I changed back to non-vfx development work my take home pay literally doubled. Obviously there is more to life than just pay! But the wage suppression has had a lasting effect... at the same time there are so many people who want into "the biz" it appears they that can/should? get away with it.
In VFX, as in game dev, there is far more supply of people desperate to get those gigs than there is demand. That's why your take-home pay doubles when you switch to something less alluring, not a competitor's agreement not to recruit.
Catmull has to look at the big picture, i.e., "Is our company going to operate well if competitors are constantly sending out emails to our employees and offering them 120% salary to switch jobs? Since it'd be equally disruptive if we did this to them and no work would ever be accomplished in this sector, let's just have a truce where we don't actively pursue one another's talent, and then we can stop destroying each others' projects with these counterproductive bidding wars."
Continuity and seniority is very important to the smooth operation of a company. VFX projects take years to complete and they're probably more sensitive to high churn than other types of projects, so the concern is even more justified in this sector than in the general sense.
If the company can't see the big picture and find a way to be productive within that climate, far worse than not getting cold calls from the recruiting department of the competitor, everyone will be out of work.
I know it's fun to hate on executives and believe me, I know they very frequently deserve every bit of it. But there is validity to the perspective that concerns for overall corporate performance must be take into the balance.
I gotta say I agree with Catmull. I understand that specifically the VFX people are trying to blow this up into some massive offense, but I simply do not see it. As I stated in other comments, this is a very common arrangement that is in no way limited to VFX, Silicon Valley, or tech.
It's great that Catmull has the backbone to refuse to apologize when someone is trying to shame him into submission. This is a surprisingly rare attribute these days.
One potential interpretation is that this artificially depresses worker salaries as workers are not continuously being auctioned back and forth. Another potential interpretation is that this allows the company to have the stability it needs to function, prevent toxic sentiment among peers who take a bid from one company or the other, potentially leaving others holding the bag for the project.
I believe this latter interpretation is the intent of most such agreements, and that the former is rarely considered legitimate (i.e., continuous bidding wars would be too disruptive to be feasible even if there were no formal agreements in place).
Such understandings are very common across competitors in all lines of work, whether they're written or not; at a former company, I was personally told by the CEO that we couldn't actively recruit someone who worked at a competitor because we didn't want to risk starting a bidding war over talent and potentially throwing everything off-kilter. Such arrangements are not SV-exclusive, let alone Pixar-exclusive, and they are done out of practicality, not malice. Market value for employees can be correctly surmised without feverish, aggressive overbidding.
The incident is frequently misconstrued as a complete block on any cross-hiring. My understanding is that it was simply an agreement forbidding cross-recruiting; a gentleman's agreement that they wouldn't try to start an arbitrary bidding war over the one company's talent if that company wouldn't try to start one over theirs. Employees were still free to seek and obtain employment at any of the major studios independently if they so chose.
I think that panicked cries of wage fixing and intentional repression of employment opportunities are not only not credible, but farcical. I'd ask myself why someone is interested in painting an imbalanced and unrepresentative picture such as that.
If anything, these agreements are a failure of the contemporary legal and HR departments across every major technology company involved in computer animation. I believe the intent of the executives was nothing more than maintaining a stable workforce. Their lawyers and HR people should have warned them that there was another dormant interpretation that could've been used by exploitative politicians to misrepresent the situation.
Last time this came up (that I saw), the Pixar, Apple, and Intel et al chiefs were being compared to Nazis. That is beyond the pale.
But Catmull in particular was involved in the more extreme implementations of the deal. In his version, not only did cooperating companies agree not to actively poach, nor to passively extend offers when approached by employees of competitors, but also they agreed to actively report amongst themselves whenever they were approached by employees of competitors.
This probably damaged the careers of engineers who were not completely satisfied with their current employer, because as an exec are you going to give a key project to someone who's about to go work for the competition?
Maybe even the Catmull version of the wage-fixing scheme has some defenders, but I think there are very few defenders of the Catmull form of the agreement compared to the fraction of HNers who are OK with the milder form that merely instructed recruiters not to actively initiate conversations with employees of competitors.
Just as we don't go out of our way to defend the guy who did rm -rf for what he did specifically, but rather move on.
IMO, the lesson to draw from this is "get better legal advice and avoid showboating politicians". Happy to move on.
EDIT: also, shortened the parent for you. I agree it was overly long.
You can't make ANY agreements to refuse to recruit from certain companies.
Making any "official" effort or policy to prevent a bid war is illegal.
Do we really believe that the big shot lawyers at all of these places (remember, many household brands were parties) are so bad they would allow a contract that was pro se illegal to be entered into, or that the executives secretly fast-tracked these documents and bypassed counsel? To me, the situation sounds much grayer than some portray it.
This case was settled, not tried. We are left to speculate on what the outcomes and conclusions would've been had the settlement not proceeded.
I don't believe their lawyers or the companies are incompetent. I believe that they were actively malicious.
They thought that the legal costs of getting caught and losing the lawsuit, or settling, would be less than following the law, so they actively chose to take the risk and break the law.
And it turns out, they were mostly correct about that. Doesn't mean that they shouldn't be condemned for it.
This is backed up by all the statements that they made about "don't put this stuff in writing! ", ect. And the fact that they stopped engaging in these practices. (if they were doing nothing wrong, then they would continue, right?)
X billion gain > Y billion loss.
These companies won. They saved more money than they lost.
From a business standpoint, it was a good idea to break the law, and get sued.
I think this is one of those occasions where it's OK for companies to individually come to this conclusion and not implement this as a practice. But the moment several companies come to a collective agreement on the same, it enters questionable and probably illegal territory.
There was an incident where I work where an employee (a new hire) set up a cron job to delete his local code tree, re-sync a new copy, then re-build it using a cron job every night. A completely reasonable thing for a coder to automate.
In his crontab he put:
rm -rf /$MY_TREE_ROOT
and as everyone undoubtedly first discovered by accident, the crontab environment is stripped bare of all your ordinary shell environment. So $MY_TREE_ROOT expanded to "".The crontab ran on Friday, IIRC, and got most of the way through deleting the entire project over the weekend before a lead came in and noticed things were disappearing. Work was more or less halted for several days while the supes worked to restore everything from backup.
Could the blunder have been prevented? Yes, probably with a higher degree of caution, but that level of subtlety in a coding mistake is made regularly by most people (especially someone right out of university); he was just unlucky that the consequences were catastrophic, and that he tripped over the weird way crontab works in the worst possible usage case. He probably even tested it in his shell. We all know to quadruple-check our rm-rfs, but we know that because we've all made (smaller) mistakes like his. It could have been anyone.
Dragging him to the guillotine would have solved nothing. In fact, the real question is "how is it possible for such an easy mistake to hose the entire project?" Some relatively small permissions changes at the directory root of the master project probably would have stopped stray `rm -rf`s from escaping local machines without severely sacrificing the frictionless environment that comes from an open-permissions shop. So if anything, the failure was systems's fault for setting up a filesystem that can be hosed so easily and so completely by an honest mistake.
The correct thing to do was (and is) to skip the witch hunt, and focus on problem-solving. I am not sure, but I think the employee was eventually hired on at the end of his internship.
For me the principle is: Standards and habits are teachable. Competence and attitude, less so. Educate and train for the former, and a failure of the former should cause you to look first at your procedures, not the people. Hire and fire for the latter.
The other side is if you play a key role (and head could roll after the hard work is done) to simply leverage that fact (perhaps with others) as an advantage such that you have get a new contract and can't be fired for X amount of time.
Commitment would be very different if people were being asked to help while some heads were rolling. Because you're a real team when everybody is going in the same direction. Any call on "people, work hard do recover while we're after the moron who deleted everything" wouldn't have done it.
You just commit to something when you know that you won't be under the fire if you do something wrong without knowing it.