shadowsun7

My two questions (a) and (b) were not rhetorical. Let’s get concrete.

a) You are advising a company to “check back after a certain period”. After the certain period, they come back to you with the following graph:

https://commoncog.com/content/images/2024/01/prospect_calls_...

“How did we do? Did we improve?”

How do you answer? Notice that this is a problem regardless of whether you are a big company or a small company.

b) 3 months later, your client comes back and asks: “we are having trouble with customer support. How do we know that it’s not related to this change we made?” With your superior experience working with hundreds of startups, you are able to tell them if it is or isn’t after some investigation. Your client asks you: “how can we do that for ourselves without calling on you every time we see something weird?”

How do you answer?

(My answers are in the WBR essay and the essay that comes immediately before that, natch)

It is a common excuse to wave away these ideas with “oh, these are big company solutions, not applicable to small businesses.” But a) I have applied these ideas to my own small business and doubled revenue; also b) in 1992 Donald Wheeler applied these methods to a small Japanese night club and then wrote a whole book about the results: https://www.amazon.sg/Spc-Esquire-Club-Donald-Wheeler/dp/094...

Wheeler wanted to prove, (and I wanted to verify), that ‘tools to understand how your business ACTUALLY works’ are uniformly applicable regardless of company size.

If anyone reading this is interested in being able to answer confidently to both questions, I recommend reading my essays to start with (there’s enough in front of the paywall to be useful) and then jump straight to Wheeler. I recommend Understanding Variation, which was originally developed as a 1993 presentation to managers at DuPont (which means it is light on statistics).

https://commoncog.com/becoming-data-driven-first-principles/

https://commoncog.com/the-amazon-weekly-business-review/

(It took that long because of a) an NDA, and b) it takes time to put the ideas to practice and understand them, and then teach them to other business operators!)

The ideas presented in this particular essay are really attributed to W. Edwards Deming, Donald Wheeler, and Brian Joiner (who created Minitab; ‘Joiner’s Rule’, the variant of Goodhart’s Law that is cited in the link above is attributed to him)

Most of these ideas were developed in manufacturing, in the post WW2 period. The Amazon-style WBR merely adapts them for the tech industry.

I hope you will enjoy these essays — and better yet, put them to practice. Multiple executives have told me the series of posts have completely changed the way they see and run their businesses.

Reality has a lot of detail. It’s nice to quote books about goals. It’s a different thing entirely to achieve them in practice with a real business.

https://commoncog.com/the-amazon-weekly-business-review/

Over the past year, Roger and I have been talking about the difficulty of spreading these ideas. The WBR works, but as the essay shows, it is an interlocking set of processes that solves for a bunch of socio-technical problems. It is not easy to get companies to adopt such large changes.

As a companion to the essay, here is a sequence of cases about companies putting these ideas to practice:

https://commoncog.com/c/concepts/data-driven/

The common thing in all these essays is that it doesn’t stop at high-falutin’ (or conceptual) recommendation, but actually dives into real world application and practice. Yes, it’s nice to say “let’s have a re-evaluation date.” But what does it actually look like to get folks to do that at scale?

Well, the WBR is one way that works in practice, at scale, and with some success in multiple companies. And we keep finding nuances in our own practice: https://x.com/ejames_c/status/1849648179337371816

I don't have his permission to tell some of these stories, but Eugene Wei has some hints of it here: https://www.eugenewei.com/blog/2017/11/13/remove-the-legend

(To be fair to you, you are adamant that SPC does not apply to software development — which I take to mean measuring the productivity or act of building software. And I think we are all in agreement there! (That said, like kqr and jacques_chester, I want to believe that this has not been sufficiently explored) But it's not true that SPC has no place in software development — one way I've used this is that because XmR charts detect changes in variation, you can use it in a customer-facing software context to see if a feature change has resulted in user behaviour change without running an A/B test. Naturally, it makes sense to have the software engineer be responsible for observing this behaviour change themselves, since XmR charts are easy enough for the layman to use, and it gives them a sense of ownership for the feature or change. Some detail (on usage vs A/B tests) here: https://commoncog.com/two-types-of-data-analysis/)

He can't make this claim (about ~half falling above/below the average line), because one of the core arguments he makes is that XmR charts are usable even when you're not dealing with normal distributions. He argues that the intuition behind how they work is that they detect the presence of more than one probability distribution in the variation of a time series.

Some links below:

Arguments for non-normality:

https://spcpress.com/pdf/DJW220.pdf

https://www.spcpress.com/pdf/DJW354.Sep.19.The%20Normality-M...

Claim of homogeneity detection:

https://www.spcpress.com/pdf/DJW204.pdf

My current simplistic (and very dumb!) solution that I've used for power-law type distributions — like HN virality, for instance — is to count the number of days between viral events, and then subject that to process control.[1] I basically take Wheeler's approach to chunky data and use that for J-curve type data, which tells me if the behaviour of my 'HN virality process' has changed.

I'd be very interested to learn of other approaches.

[1] HN traffic for commoncog.com displays routine variation most weeks with an Upper Process Limit of 192 and a Lower Process Limit of 0, unless one of my articles hit the front page, at which point I get 11-16k additional uniques).

(From a brief reading of this thread, it seems like kqr, jacques_chester, and I are the only ones who have put this to practice in non-manufacturing contexts — though correct me if I'm wrong.)

The bulk of the debate in this HN thread seems to be centred around what is or isn't a 'stable process'. I think this is partially a terminology issue, which Donald Wheeler called out in the appendix of Understanding Variation. He recommends not using words like 'stable' or 'in-control', or even 'special cause variation', as the words are confusing ... and in his experience lead people to unfruitful discussions.

Instead, he suggests:

- Instead of calling this 'Statistical Process Control', call this 'Methods of Continual Improvement'

- Use the term 'routine variation' and 'exceptional variation' whenever possible. In practice, I tend to use 'special variation' in discussion, not 'exceptional variation', simply because it's easier to say.

- Use the term 'process behaviour chart' instead of 'process control chart' — we use these charts to characterising the behaviour of a process, not merely to 'control' it.

- Use 'predictable process' and 'unpredictable process' (instead of 'stable'/'in-control' vs 'unstable'/'out-of-control' processes) because these are more reflective of the process behaviours. (e.g. a predictable process should reliably show us data between two limit lines).

Using this terminology, the right question to ask is: are there processes in software development that display routine variation? And the answer is yes, absolutely. kqr has given a list in this comment: https://news.ycombinator.com/item?id=39638491

In my experience, people who haven't actually tried to apply SPC techniques outside of manufacturing do not typically have a good sense for what kinds of processes display routine variation. I would urge you to see for yourself: collect data, and then plot it on an XmR chart. It usually takes you only a couple of seconds to see if it does or does not apply — at which point you may discard the chart if you do not find it useful. But you should discover that a surprisingly large chunk of processes do display some form of routine variation. (Source: I've taught this to a handful of folk by now — in various marketing/sales and software engineering roles —and they typically find some way to use XmR charts relatively quickly within their work domains).

[Note: this 'XmR charts are surprisingly useful' is actually one of the major themes in Wheeler's Making Sense of Data — which was written specifically for usage in non-manufacturing contexts; the subtitle of the book is 'SPC for the Service Sector'. You should buy that book if you are serious about application!]

I realise that a bigger challenge with getting SPC adopted is as follows: why should I even use these techniques? What benefits might there be for me? If you don't think SPC is a powerful toolkit, you won't be bothered to look past the janky terminology or the weird statistics.

So here's my pitch: every Wednesday morning, Amazon's leaders get together to go through 400-500 metrics within one hour. This is the Amazon-style Weekly Business Review, or WBR. The WBR draws directly from SPC (early Amazon exec Colin Bryar told me that the WBR is but a 'process control tool' ... and the truth is that it stems from the same style of thinking that gives you the process behaviour chart). What is it good for? Well, the WBR helps Amazon's leaders build a shared causal model of their business, at which point they may loop on that model to turn the screws on their competition and to drive them out of business.

But in order to understand and implement the WBR, you must first understand some of the ideas of SPC.

If that whets your interest, here is a 9000 word essay I wrote to do exactly that, which stems from 1.5 years of personal research, and then practice, and then bad attempts at teaching it to other startup operator friends: https://commoncog.com/becoming-data-driven-first-principles/

I don't get into it too much, but the essay calls out various other applications of these ideas, amongst them the Toyota Production System (which was bootstrapped off a combination of ideas taught by W Edwards Deming — including the SPC theory of variation), Koch Industries's rise to powerful conglomerate, Iams pet foods, etc etc.

As an update, I relocated to a different city and spent four months training Judo for 5 hours every day and have since reached the vocab point for the sport. That experiment was covered here: https://commoncog.com/expertise-acceleration-experiment-judo... and https://commoncog.com/mental-strength-judo-life/

I'm looking forward to trying Discourse chat — if the level of engineering and thoughtfulness is the same as the rest of the feature-set, it'll be a great addition to my community.

Well, it seems you're drawing on the same argument that has already been addressed (or at least mentioned in passing) in the article.

So ... pop quiz: how is reading for concept instantiations different from reading for patterns?

> the High Court found that the appellant had borderline intellectual functioning; not that he was suffering from mild intellectual disability. This was conceded by the appellant’s own psychiatrist, Dr Ung Eng Khean (“Dr Ung”). Further, Dr Ung also accepted (see Nagaenthran (CM) at [76]) that borderline intellectual functioning is not a mental “disorder” as set out in the American Psychiatric Association, Diagnostic and Statistical Manual of Mental Disorders (American Psychiatric Association Publishing, 5th Ed, 2013). Further, in Nagaenthran (CA) (at [34]–[41]), we held that even assuming the appellant suffered from an abnormality of mind, any such abnormality did not substantially impair his mental responsibility, because he did not lose his ability to tell right from wrong.

I then think Nagaenthran's lawyer made a grievous concession:

> [The appellant’s counsel, Mr Thuraisingam] eventually conceded that this was a case of a poor assessment of the risks on the appellant’s part. But, as the Minister stated in Singapore Parliamentary Debates, Official Reports (14 November 2012) vol 89 … ‘[g]enuine cases of mental disability are recognised [under s 33B(3)(b) of the MDA], while, errors of judgment will not afford a defence’. To put it quite bluntly, this was the working of a criminal mind, weighing the risks and countervailing benefits associated with the criminal conduct in question. The appellant in the end took a calculated risk which, contrary to his expectations, materialised. Even if we accepted that his ability to assess risk was impaired, on no basis could this amount to an impairment of his mental responsibility for his acts. He fully knew and intended to act as he did. His alleged deficiency in assessing risks might have made him more prone to engage in risky behaviour; that, however, does not in any way diminish his culpability.

I'm not sure if your reading of 'intellectually disabled man' aligns with the court's reading, and I don't think it's as clear cut as "oh, Singapore executed a man with no mental responsibility over what he is doing, according to the rule of law there, and so LHL is evil"

> Malaysia currently has a moratorium on the death penalty, due to serious national debate (in which the government has been an active participant) on whether to abolish it, or at least significantly narrow its scope

You're right on the moratorium. 'Due to healthy debate' is a rather charitable reading, though. There's a depressing account of the entire moratorium in Chapter 45, 'Law Reform', from former AG Tommy Thomas's autobiography My Story: Justice in the Wilderness, which lays out the background machinations of the former administration's attempt to repeal the death penalty, which was ultimately a failure. The moratorium was implemented by the Prisons Board under the instructions of acting AG Engku Nor Faizah Engku Atek, and supported by Thomas. More surprisingly, (if Thomas is to be believed) the Prisons Board themselves had no objection to abolishing the death penalty! I won't try to summarise the complex political machinations here, but saying that 'strong National debate' is a factor for the moratorium would not be Tommy Thomas's reading of the situation. I don't expect the moratorium to last — though I pray that it will. But it's difficult to say which side might use it as a political football given the current state of Malaysian politics.

My overall point: using the moratorium on the death penalty as an example of Malaysia's 'enlightened approach' vs Singapore is grossly mistaken; it's more accurate to describe it as a political football used to score points against one's opponents (made complicated by the fact that there is quiet support for the penalty from both the conservative Malay power base as well as the conservative Chinese political power base, on both sides of the parliament, as Thomas found out the hard way). I say that the chapter is depressing because 'national debate' seems to have very little to do with it.

Nagaenthran's lawyer disagreed with these assessments, but, unable to find a psych willing to certify Nagaenthran's intellectual disability, got so desperate that he signed an affidavit declaring himself convinced that Nagaenthran was intellectually disabled. The court rejected this affidavit after establishing that Nagaenthran's lawyer had no medical expertise and after he admitted that he was essentially 'speculating on what the appellant's mental age was'. Then the AG's office got a fourth medical assessment in 2021, which was blocked from submission to the court by Nagaenthran's lawyer, arguing that the 'appellant was interested in 'medical confidentiality'. At that point the court sort of threw up its hands and went 'you claim he's intellectually disabled but now you want to block medical reports on that exact issue, due to privacy reasons?'

Keep in mind that at this point, the case had been in Singapore's courts since 2010, and the various assessments and appeals had lasted more than a decade.

Nagaenthran's family, with the help of activists, requested appeals from both Malaysian's prime minister and a direct appeal to Singapore's president, who had the power to stay the execution by issuing a pardon. Singapore's president said she looked into it and was satisfied that the case was sound.

Finally, Nagaethran's family — again with the help of activists — filed a final appeal, arguing that the sentence was unconstitutional because Chief Justice Sundaresh Menon, who presided over Nagaenthran's previous failed appeals, was also the Attorney-General during his conviction.

At this point the panel of three judges threw the application out on the grounds that it was a 'calculated attempt' to diminish the finality of the court process. "No court in the world would allow an applicant to prolong matters ad infinitum" by filing such applications, said the judges. They'd been pushing such applications over the course of a full decade by that point, switching arguments three times.

Source: https://www.channelnewsasia.com/singapore/nagaenthran-dharma...

And read the full judgment: https://elitigation.sg/gd/s/2022_SGCA_26

I'm sympathetic to the goals of the activists — they want to push a test case to repeal the death penalty. The problem is that it doesn't seem to be a particularly good test case. Nagaenthran had successfully stayed his execution multiple times over the past decade, and he had pretty decent legal representation throughout. (Notwithstanding the affidavit — which probably harmed his case more than it helped).

You may argue that Singapore is a 'rather extreme outlier among developed countries' — but its policies on drug enforcement is 100% consistent with just about every other country in its vicinity. In fact, had Nagaenthran been caught in his home country of Malaysia, he would be sentenced to death just the same, since Malaysia has pretty much the same laws on drug trafficking. The activists have been raising international awareness of the case, but Malaysia's government is pretty blase about the whole thing, since they have similar laws and a similar justice system, and Malaysian anti-narcotics officers have a long history of working with Singaporean anti-narcotics officers to stop the drug trade from trickling into their countries from the Golden Triangle to their north.

Edit to add: I was actually prepared to get angry over this case, but then I read the full judgment and now I think ... "hmm, this is a more complicated case than I thought."

I’ve updated the bottom of the post to link to the many other articles I’ve written on expertise. These are either summaries of papers, or summaries of branches of the research, or pointing out obvious holes in the popsci representations of the research, or synthesise various (mostly military-funded) approaches to accelerating expertise.

This particular essay is a note from personal application.

The article you link to is essentially a response to 2 pages in the book, where Hoffman et al mention, almost in passing, that CLT is a silly theory when you want to train for real world scenarios (the intuition is that if you’re training marine fire squad commanders to plan on the battlefield, perhaps it helps to simulate shooting at them during training?) Hoffman et al use this as an example of a learning theory that doesn’t seem to map to real world requirements.

This reads like a disagreement over one particular dismissal in the book, perhaps because CLT is a pet theory of the article’s authors. The problem: this argument is not core to the book!

The article does not, for instance,

a) Deal with the many examples of successful real world accelerated training programs with no curriculum design (as is commonly understood; ordering of simulations isn’t really designing a syllabus) in Chapter 9 (some of which were designed by some of the authors)

b) Have a rejoinder to the two learning theories presented in Chapter 11 that the authors claim underpins their training approach (if there were something to attack, this would be it!)

c) Nor have a rejoinder to a more central claim in the book, (and to my mind a more controversial claim) that atomisation of concepts impedes rapidised training.

And, perhaps most surprisingly to me, your claim that

> Reading it I got the distinct impression that the authors did not understand a great deal of the research they cited, either when supporting or dismissing it.

is remarkable, given that one of the authors of Accelerated Expertise is Paul J Feltovich, one of the founders of the field of expertise research, and a contemporary of Ericsson’s.

The basic idea is this:

1. Deliberate practice only works for skills with a history of good pedagogical development. If no such pedagogical development exists, you can’t do DP. Source: read Peak, or any of Ericsson’s original papers. Don’t read third party or popsci accounts of DP.

2. Once you realise this, then the next question you should ask is how can you learn effectively in a skill domain where no good pedagogical development exists? Well, it turns out a) the US military wanted answers to exactly this question, and b) a good subsection of the expertise research community wondered exactly the same thing.

3. The trick is this: use cognitive task analysis to extract tacit knowledge from the heads of existing experts. These experts built their expertise through trial and error and luck, not DP. But you can extract their knowledge as a shortcut. After this, you use the extracted tacit knowledge to create a case library of simulations. Sort the simulations according to difficulty to use as training programs. Don’t bother with DP — the pedagogical development necessary for DP to be successful simply takes too long.

Broadly speaking, DP and tacit knowledge extraction represent two different takes on expertise acquisition. For an overview of this, read the Oxford Handbook of Expertise and compare against the Cambridge Handbook of Expertise. The former represents the tacit knowledge extraction approach; the latter represents the DP approach. Both are legitimate approaches, but one is more tractable when you find yourself in a domain with underdeveloped training methods (like most of the skill domains necessary for success in one’s career).

One way to pick who to pay attention to is ‘believability’ — meaning that you listen to those with at least 3 successes, and a coherent explanation when probed.

Pointing out survivorship bias is a common rejoinder to this view. But when you’re trying to put things to practice (not get at some universal truth like a researcher would) you often cannot wait for perfect samples. So you pick certain practices from believable people and test them against reality, and then hold the lessons loosely, making sure to update based on further experiments (which is necessary because life is messy and full of confounded variables).

Over time, it becomes clearer what is useful and works for you, and what isn’t and is perhaps a quirk of the other organisation’s context. But I’m not saying anything new; this is how we learn from life.

Related: Brian Lui’s loose feedback loops https://brianlui.dog/2020/05/10/beware-of-tight-feedback-loo...

And the problems of learning from experience: https://commoncog.com/blog/the-hard-thing-about-learning-fro...

I have not included any such theories because I have not found them useful. The links are to say something simple: my entire epistemology is pragmatic. That is, true knowledge should lead to effective action. If I cannot apply it and get results, then it is useless to me, and irrelevant when writing up notes for other practitioners.

I have noticed that your claim is that 'here is some theory that is well established and rigorous and old'. I have also noticed that your claim is not 'here is some theory that I have found useful, and <insert notes from application>.' I pay attention to arguments of the latter form, because it usually indicates something that I can integrate into my practice. Because you have not included notes from actual application, I am not particularly interested in your argument.

(But if you can provide an applied account, I’m all ears!)

That is not to say that org theorists are useless, or that research is useless. I have found Herbert Simon's work on organisational decision making useful as a lens on practical rationality, for instance. I’ve also spent a lot of time digging into expertise research for applied ideas.

I think the bar I use is simple: when reading a theory, I ask myself if there are actionable handles. If so, I tend to pay attention. If not, or if I’ve applied it and it doesn’t work particularly well, I discount the research.

That said, I have noticed that good organisational builders — with a track record of actually building orgs — say different things from organisational theorists. And I think the reason for why is interesting: I suspect that org builders are interested in actual org outcomes, while org theorists are too far removed from actual application; they are interested in tenure.

(Edited to soften tone.)

I would ask the OP a simple question: what organisations have they built, and where are they now?

Longer, but establishes the epistemology of the blog: https://commoncog.com/blog/practice-as-the-bar-for-truth/ and https://commoncog.com/blog/four-theories-of-truth/

If you want to read books to prepare yourself, read:

- Sources of Power by Gary Klein

- Power of Intuition by Gary Klein

Before AE.

And then be prepared to download LOTS of papers to read concurrently while reading the book. AE is not written for the layperson.

This body of knowledge is built up over time, usually through trial and error. It is then passed from coach to player and coach to coach.

The dirty little secret of DP is that it cannot be done in domains where no good pedagogical development exists. This is a definitional thing. You may read more about it in Peak, Ericsson's popular science book on the topic (summary: https://commoncog.com/blog/peak-book-summary/)

I want to repeat this, so it is very clear: you cannot do DP in a domain where there is no good pedagogical development.

So: rock music, jazz, some aspects of computer programming, software architecture, marketing, leadership, management — all of these are domains with under-developed pedagogical development. Don't get me wrong — there are experts in all of these domains. But they got there through trial and error and reflection, not DP (see: cognitive transformation theory for an explanation of how that occurs here: https://commoncog.com/blog/the-hard-thing-about-learning-fro...)

If you want to do DP, you would have to come up with training techniques for all of these skills, and test them over a long period of time. In other words, you would have to do pedagogical development yourself before you can do DP.

One other implication: anyone who says "oh, just go do DP" is likely someone who has not a) looked at the research closely, or b) not actually tried to put it to practice.

Applying DP is a lot harder than you might think.

Anyway, once you understand this, you may now pursue two different lines of inquiry:

- How do good coaches do pedagogical development? I recommend John Danaher to give you a taste: https://youtu.be/ktuw6Ow4sd0?t=2396 (on developing new techniques) and https://www.youtube.com/watch?v=ktuw6Ow4sd0&t=6211s (on his creation of the leg-lock system)

- How do you develop training without pedagogical development. And for that I recommend my tacit knowledge series (https://commoncog.com/blog/the-tacit-knowledge-series/) and my summary of Accelerated Expertise (https://commoncog.com/blog/accelerated-expertise/)

I've actually been putting some of the CTA methods mentioned in the Oxford Handbook to practice! See: https://commoncog.com/blog/john-cutlers-product-expertise/ for one accounting; I'm currently trying it out with an international Judo coach right now, on technique analysis. (We can't meet due to COVID, since I cannot fly to him, so what I want is to extract his ability to perform video analysis, so that I can perform it on myself and on other competitors).

If I were to summarise why the CTA/Oxford Handbook approach is more tractable, it is this: you don't 'stumble' into a way to deliberately practice something — you do 'pedagogical development'. But pedagogical development is hard! One of the things that I've found most perplexing about Ericssons's work is that, in Peak, he provides an accounting of a skill domain (memorisation competitions) that experienced pedagogical development during his time. Thanks to that pedagogical development, DP techniques could be used in the domain. But he spent no time talking about the pedagogical development!

So I've concluded that it's simply too hard to do good pedagogical development. Better to adopt CTA methods, which already work, and use those to design training programs for myself.

And while it's too early to tell (check back in a few years) I think it's more fruitful.

Not paying attention to research with military provenance seems rather silly. What matters more are the questions: does this work? Where has it been tried? And can I apply it to my company or career?

In sum: DP simply has too high a bar to clear. Want to do DP? Well, has there been a few decades of pedagogical development in your field so that you can be coached like with chess or math or tennis? NO? Ok, you're tough out of luck. No DP for you!

So the real question is: if you are working in marketing or management or leadership or computer programming, all of which are domains that matter to our careers, these domains do not have good pedagogical development — so how do you get good?

The answer to this, at least from one branch of expertise research, is to extract mental models of expertise from the heads of domain experts, and then turn those into training programs. This is a far more tractable training approach. See: https://commoncog.com/blog/accelerated-expertise/ for a summary (note: Paul Feltovich is one of the founders of the field of expertise research, and a contemporary of Ericsson; he's one of the authors of the book).

For a full survey of the approach, see the Oxford Handbook of Expertise https://www.oxfordhandbooks.com/view/10.1093/oxfordhb/978019... (as opposed to the Cambridge Handbook, which is the Ericsson school.)

1. The Tacit Knowledge Series (which argues that methods from Naturalistic Decision Making are more useful when you are in a skill domain with no good pedagogical development) https://commoncog.com/blog/the-tacit-knowledge-series/

2. Accelerated Expertise — which covers methods from the military to accelerate expertise acquisition (and, yes, some of it uses DP principles, but it isn't really DP) https://commoncog.com/blog/accelerated-expertise/