A/B and telemetry-worship reminds me of something I posted on Fedi earlier. Pasting it below:
---
"Guys, I have an idea. Let's make a browser optimized for the kind of people who don't opt out of telemetry, and market it to privacy-conscious people." -- Top minds at Mozilla Corp
If you market to privacy-conscious people and you base all your decisions on data from telemetry, you're gonna get a very skewed perspective. You'll see disproportionate representation from fans (people who want to enable telemetry because they love Mozilla and want to help, and will rationalize any decision Mozilla makes) and very non-technical users who don't know what telemetry is and how opt-in/out works. Technical users who disable stuff like telemetry, analytics, etc. are going to look like a rounding error.
This explains a lot of Firefox's changes.
As such exactly what it is, will depend on company/industry/market/team/project/etc. It may be revenue or units sold or widgets made or turnover or customer satisfaction or days without accident etc. In principle, it communicates to teams what is important to business and helps everybody focus and sync.... with usual real-world caveats and implementation risks.
I imagine here, for a large-company linux team, a "KPI" was "how many commits you made" - they measure it, they track it, and reward/recognize/punish team members based on it. So people start working toward increasing the KPI in any feasible way - if you only want me to increase the KPI of commits done, OK, fine, I'll do some commits.
This is a great explanation, and I just wanted to nitpick that for some KPI's (e.g. percentage of defective components, or latency), you would aim to "decrease" the KPI, and in some rare instances, you'd want to keep the KPI within a range (e.g. resource utilization, or inventory size, when you want a predefined amount of buffer).
It's a shit system that does exactly the opposite of what it's supposed to do IMHO, but that kind of stuff doesn't belong in an ELI5 explanation :-D.
The issue isn't whether KPIs are good or bad but what you do with them.
KPIs don't need to be tied to actual people, usually they are tied to teams, systems, projects, processes, etc.
For example, a number of users resigning from our services might be a KPI.
KPIs are an important tool to understand what is going on and whether we are going in the desired direction or not and to communicate this to your teams.
KPIs are a tool for telling your organization what you think is important, to set the incentives in the right direction.
KPIs will not help you if you have no effing idea how to set incentives -- they are just a tool in case you know what you are doing.
Because they are just one step from being a target for your teams, you need to be very careful to communicate what you mean exactly when you set KPIs.
Now, if you make number of users resigning from your services a KPI, you immediately need to set some additional guarantees that this is not overused -- for example by making it difficult for your clients to resign. This might be another KPI (user satisfaction with regards to how easy it was for them to resign).
Humans are extremely good at gaming KPIs, and will do so as long as they're rewarded for it.
Organizations have hard time improving without measuring their performance and communicating incentives.
There isn't magical solution you can scribe on a paper and tell everybody -- this is exactly what you need to do to achieve success.
The best what you can do is compromise, it is unavoidable.
So we know setting targets can make wrong incentives. It is also not a reason to not be setting targets. It is a reason to make sure you damn set those incentives right and take close look that you are getting what you have intended.
Metrics are either (1) targets or (2) things that you are trying to analyze in relation to the metrics that are targets or (3) a waste of the time you spent gathering them.
The reason metrics are often bad when they become targets is the metrics arr usually not actual direct measures of goals, but things assumed to be convenient proxies, but when you push hard on optimizing them, they stop being good proxies because as well as being easier to measure than the real objective, once you pick any low-hanging productive fruit they are inevitably easier to improve in ways that don’t improve the real objective as much as the proxy (or at all, or which negatively impact it.)
You have a target that you can't measure directly, hence you measure a proxy. That only works for as long as people don't game it and aim for the proxy instead of the actual target. Also, in addition, at least in research every statistician worth their salt knows that their proxy is not the actual thing and only their best effort at approximating it, hence in good studies, limitations of the methodology are extensively discussed.
Importantly, the moment you realise that your proxy is being gamed, you need to switch proxy.
I think this is only one example of this annoying trend of pretending to be "data-driven" by coopting numbers and fancy statistics without also adopting everything we know about uncertainty and how we can quantify it. "Data" in many businesses often implies a level of clarity ("look, the graph always goes up") that often does not actually exist and I suspect that this can be hard to understand for certain types of managers.
Output metrics chosen correctly also tend to represent things that the team/person has a reasonable degree of control over. For example, a developer KPI probably shouldn't be number of new customers because that's something they have vanishingly little control over, especially at an individual level.
The problem is that those things that a number of different teams are contributing to are probably the thing that the company cares about.
If there's a single KPI and all the reward is tied to that without any balance, sure that will be gamed.
But if there is a well-defined, appropriately complex reward function aligned with the utility of whatever the organization delivers to the outer world, I would consider that a positive.
The original quote is
"Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes."
I don't think there is a a good solution. Maybe with radically shortened work hours and less pay disparity, the strives can strive off the job instead.
This isn't just theory. This is precisely why there are laws that prohibit rewarding people based on how they voted.
> This is precisely why there are laws that prohibit rewarding people based on how they voted.
Voting is not an evaluation method of the voter: the voter is the one doing the evaluating. Because the powers should not discriminate between voters as you say, the vote cannot be used to evaluate the voters on and individual basis at all.
Maybe the answer is that: no individual/team assessments of any sort. That makes no incentives. But that also doesn't help with fine-grained strategy.
...by not tieing incentives to it? I feel like I must not understand your question because this seems blindingly obvious...
> Voting is not an evaluation method of the voter: the voter is the one doing the evaluating. Because the powers should not discriminate between voters as you say, the vote cannot be used to evaluate the voters on and individual basis at all.
I don't even get where you got this strawman from.
> no individual/team assessments of any sort. That makes no incentives. But that also doesn't help with fine-grained strategy.
No, just dont use KPIs that are critical to understanding your business in assessments tied to incentices because that will distort your KPIs and make you less capable of understanding your business.
Mondragon please get in the tech biz.
When there as many evaluators as evaluatees, no standard evaluation procedure is needed on the theory that all the different ways and biases will cancel out. That doesn't work over time, as clearly America's electorate gets interested in different things, but I am less worried about that for co-ops.
The hope for co-ops vs state democracy is two fold.
Firstly, because people do work many hours at their job, but don't necessarily spend any time running the state, I hope they are more informed and engaged.
Secondly, the "both options are bad, I hate this" problem should be addressed by A, proportional representation (just like should be done with state democracy states), but also B of switching jobs. The barrier of switching jobs is much lower than immigrating, which ought to more than any thing else reign in defeatist apathy. And Co-ops can still fire people, remember.
/s
Right, but my post, the post I was responding to, the post the post I was responding to was responding to, and the article in the link, was specifically about a situation where KPIs are tied to actual people. I'm not sure what you're arguing for or against.
Use SMART objectives (Specific, Measurable, Achievable, Relevant, Time-bounded) and set a realistic service uptime (e.g. how many nines you want in a given month/quarter/whatever) and KPIs directly enable good decisionmaking at all levels of the org.
Manager up in your grill about a choice you made? Point at the KPIs and say "this will help make it easier to meet our objectives for KPI 1.1" or whatever.
Engineer's performance is measured, in part, by number of kernel commits. So, the number of kernel commits is a key performance indicator for the developer.
These organizations pushing trivial/garbage patches and wasting maintainers' time absolutely have to be called out by the community as unacceptable, so I'm glad to see the post. This is a massive waste of time, time which open source contributors/maintainers often are already short on.
Now, if the people approving patches are just normal employees who can get fired if a kernel sponsor complains enough, it’s going to be much harder for them to successfully deflect all the bullshit without ever caving in.
There has been a longstanding problem of people submitting low-quality patches to maintainers - not to Linus personally - and that problem predated Linus stepping back for one release.
Linus was not ousted and continues to have nearly absolute control; he has, for decades, chosen to exercise that control by delegating ownership of parts of the code to other people. Those people receive and review patches and incorporate them into their repos, which Linus pulls in bulk. (This is the origin of the term "pull request," which predates GitHub's use of it.) He quickly reviews those patches to make sure there's nothing weird, but he generally trusts that his "lieutenants" are making good decisions.
That means that the time being wasted here is the time of individual maintainers, who are reviewing the patch, making a full judgment on it, and pulling it into their tree. Linus sees all such cleanups once per cycle and looks at them fairly quickly.
Almost all of those maintainers have been "normal employees" of various companies for many, many years. Again, take a look at the MAINTAINERS file.
Linux health happened *despite* Linus' manners, not because of them.
For one, I think it might be possible Linus's no-nonsense attitude is for the better of the organization as people who can't work with it are leaving causing Linux developers, on average, to be more no-nonsense and also able to coexist better with other no-nonsense people.
But, it is my conjecture only. We would never know how Linux would fare if Linus was another guy.
Wasn't the other person doing exactly the same, just on the "pro-asshole" side of the argument?
These are logically not the same structure of argument. I'm not making any character judgements about people involved in this discussion, or on the LKML either for that matter.
I have trouble understanding why it bothers people so much that Linus is rude sometimes. You don't have to interact with him if you don't want. You can even contribute to the kernel without interacting with him. Whatever he's been doing has been working for going on 3 decades now, I don't see a burning need to change it because it bothers some outsiders.
There's this weird sense of entitlement. People want to elbow their way into this long-running and successful project and start telling everybody how they ought to behave. They think they have the right to contribute on their own terms without bothering to understand the project culture, and think it's everybody else's responsibility to make things easy for them and behave in a way they approve of.
If you really think the LKML is too toxic and hostile, to the detriment of the project, then feel free to fork it and start up a parallel project without the problems you see. Surely your friendly corporate-style culture will attract more contributors and soon you'll have the more successful kernel, right?
Personally, I like having somebody in charge of the core of the OS who's so concerned with correctness and hygiene that he gets upset when they're violated.
>feel free to fork it and start up a parallel project without the problems you see
Most Linux distros are basically already doing this. They all have their own patchsets. It's well beyond correctness and hygiene at this point, if you actually look at the changes that are being disputed, it already falls a lot more in the "cultural differences" category.
The cultural differences that cause distro forks are debates over free-vs-libre, what belongs in the kernel vs userspace, and standardization issues, not (afaik) "we won't submit this to the upstream kernel because we don't approve of the way they talk to each other there". And I think it's very important for the core kernel devs to hold the line on those issues.
Can people actually point to instances where he was rude?
I'd agree with this for 99% of the time. There's a small 1% of the time though...where sometimes people really need to be told to fuck off.
On GPLv3 lots of calls for Linus to stop supporting GPLv2
"How do we get you to stop..."
https://www.youtube.com/watch?v=PaKIZ7gJlRU
And the list goes on and on. That said, his decisions seem to have worked out amazingly well. At some point I can imagine getting tired at having to repeatedly defend your views.
As someone who regularly deals with bugs and inconsistencies in Linux, I wouldn't say so. For me personally, I just want to be able to fix the issues, and would rather not spend the time bickering with someone who has some kind of personal vendetta about something that I don't know about. I try hard not to dump my baggage on other open source maintainers, I hope others can do the same.
It works for Linus because of special circumstances that are not easily replicated.
First, Linus is the creator of the project. He is also competent AF (borrowed your jargon here).
Second, people understand Linus is doing this solely out of care for the project. If you look at history of his outbursts you will notice a pattern.
Third, Linus tends to be right.
Fourth, Linus has and had since almost very beginning celebrity status and people tend to approve of eccentric behavior of celebrities more.
Linus disagrees with you, so strongly that he invested a great amount of his time, reputation, and political capital in changing his behavior and that of his organization.
https://news.ycombinator.com/item?id=18000698
I think I'll trust Linus on that. Linux seems pretty successful.
My question remains: What is the basis of that? Do you work with him? Read his biography at least (assuming there is one)?
But mostly, maintainers want to just focus on their work and not be bothered by corporations and their developers trying to game the system just to prop up their position.
Could you give an example of a non trivial patch which value would be unfeasible to assess objectively? I am not familiar enough with the maintenance process to understand how hard it is to measure objectively the quality of a patch.
For starters there isn't even an objectively established definition of value. You can't objectively measure something that only has subjective definitions.
In a free market you could say that the price could be proxy for the value (ie how much somebody can pay for it should at least theoretically correspond to its value for that person). But how you do that for an open source project?
The advice to companies using this KPI would be: let your own dev managers evaluate the quality of the commits of their engineers, and rate them on quality as well as quantity, then submit. Engineers will quickly stop creating busywork when their managers get wise to it.
Given that it’s annoying enough that we have articles and comments on it, perhaps investing maintainers time into developing metrics that work for the maintainers isn’t a terrible idea.
You can totally build something that is a decent metric, but as soon as you create incentives to "game it" (optimize for the metric not the actual goal you're trying to measure), it will be gamed, and creating metrics that are resistant to that is nearly impossible in most cases.
Why not just keep commits as a metric, then estimate the quality of those commits by sampling. If the organization appears to be gaming the stats then flag it as such.
I do agree that tracking commit counts isn't very useful but tracking commit frequency, in my opinion, is quite useful. I'm obviously biased but I use commit frequency to tell me how fast a project is moving and what is the investment.
Take the following for example:
https://public-001.gitsense.com/insights/github/repos?p=impa...
by breaking down how frequently contributors commit on a daily basis, you can get a very good sense of investment and speed. In the case of the vscode project, Microsoft is investing a lot of resources into vscode and it is evolving at an extremely fast pace.
Software metrics in general isn't the issue, it's the lack of context around them that is the issue.
Disclaimer: I'm the creator of the tool that I'm linking to
This is one of those ideas that I think should totally be a thing, and yet I think wouldn't work and thus shouldn't be a thing.
People will work for the carrot, so give carrots for the work maintainers want to see.
In some companies, the amount of patents or Linux kernel patches you get accepted is direct measure of your success.
As you know, whatever you measure becomes a target -- these guys feel very pressed to get ANY kernel commits accepted, no matter how small, irrelevant or nonsensical. Because in the very end almost nobody sees the amount of value that a company creates for Linux users. What everybody is looking at is that "Company X is nth on the list of biggest kernel contributors".
So "KPI grabbing" is meant to mean the practice of not caring for the value or quality of your contributions but rather for the count of the commits or LoC you can get accepted.
This is damaging to Linux kernel developer community because it just creates work for maintainers without creating much or any value at all.
These maintainers are absolutely critical resource for the project and their number and availability translates directly to how much stuff can actually be done. So no wonder people are irked by it.