Perhaps also Chesterton's fence [2].
Maybe also the whole premature-optimization thing [3].
And of course the too-clever-by-half coyotes [4].
Really maybe it just comes down to "be wary of making changes that reduce resiliency."
I was hoping to come up with something cohesive with this comment, but really I guess I just agree with what you say.
And I think there are a bunch of people sort of circling around the same idea, which I don't think we've really quite landed on a precise definition of, just as you say.
[1] https://en.wikipedia.org/wiki/Antifragility
[2] https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton's_...
[3] https://en.wikipedia.org/wiki/Program_optimization#When_to_o...
Thank you for this link. I'm halfway through that article and will probably read every single one on that website.
He uses the analogy of 'too clever by half' to exemplify his idea that 'financial innovation is always wrong'.
Nothing could be further from the truth. Insurance products have changed the world just as much as any technical innovation.
Mortgaged backed securities are not a bad thing, far from it, they allow more efficient use of capital by having 'saving Germans and Japanese' invest their money where they otherwise would not be able to.
The problem in the 2008 crash was soft systematic corruption and un-ironically a lack of fragility (ie one bank goes down it takes the rest down like dominos) - not necessarily the securities themselves for which he didn't actually even provide any basis of his negative assertion.
Efficiency is usually how we gain productivity and it's borderline absurd to say there is inherently something wrong with it on the whole. Like anything 'it depends'.
If you can have a software algorithm outperform 100 analysts on weather predictions for your fleet of drivers ... that's probably efficient. But cutting down operating margins so that any bump in the economy will leave you flat is maybe 'over optimisation'.
I don't really have a horse in this race, but I think you're misreading the article.
He says: "Every truly disruptive discovery or innovation in history is the work of coyotes. It’s always the non-domesticated schemers who come up with the Idea That Changes Things. We all know the type. Many of the readers of this note ARE the type."
That's not a criticism, that's a point of praise.
He then follows it up immediately by saying: "Financial innovation is no exception. And this is Reason #1 why financial innovation ALWAYS ends in tears, because coyotes are too clever by half. They figure out a brilliant way to win at the mini-game that they’re immersed in, and they ignore the meta-game. Eventually the meta-game blows up on them, and they’re toast."
That isn't saying it's a bad thing, it's saying that the people who come up with the new ideas lose sight of the broader picture and get taken out by "the thieving raccoons" and the State.
He's saying "the coyotes" lose sight of the broader picture, just like in the famed XKCD [1] where the "too-clever-by-half" computer person encrypts all their data, and forgets that the thug who is going to come looking for it will just beat the piss out of them with a wrench until they turn over the key.
The core nugget of the article, in my opinion, is exactly the "meta-game is what always gets you" aspect.
It's the same thing that NN Taleb refers to as "2nd order effects."
Could be a Chesterton’s Fence scenario
Instead of bemoaning efficiency, it’d be interesting to reward/value redundancy and antifragility, at least at the system level.
I think this could mean trust busting, regulation, and general cultural shifts.
There was zero redundancy versus leaving the directory open so they could open the next file (or using the application's "Open File" dialog).
That is a perfect example of wasteful motion (in their case due to a poor mental model of how computers worked, as I learned through later discussions) that could be simplified significantly without loss of quality or redundancy in the system.
Contrast this with: The surgical office called me this morning and stated, "The surgery is for a ganglion cyst on your left wrist." Which I confirmed. When I go in on Tuesday for the surgery this will be repeated, and a mark will be made on the area to be cut open (though in this case it'd be really hard to screw up and open the right wrist, as there is no, quite visible, cyst there). That is useful redundancy of the sort you describe. Removing any step (the initial visit a week ago, the call today, the check when I arrive, the mark on the wrist) and you increase the risk of error.
Punch line - Sausages coming from a new modern factory didn't taste the same. The new, more efficient building removed a long transportation step where the partially finished sausages picked up flavors and scents coming from different parts of the factory. They had to create a new process to manually add those flavors that they were accidentally getting for free from the old factory layout.
It would be good to have proof of a Chesteron's Fence analysis to say "yes, we know that Y & Z purposes and have analized the cost/benefit to removing them and the populations/systems impacted" - would this be an impact analysis?
In other scenarios, the process fails if any of the steps fail. In that case, redundancy is less stable, and you can improve both stability and efficiency by eliminating unnecessary steps.
In either case, there may be other considerations involved as well (flexibility, visibility, recoverability...) but sometimes we just didn't see a better way to do something.
This type of example exists in all industries. For example, finding a new alloy that has strictly superior properties across all dimensions for a specific use case. Or upgrading mail delivery routes using better pathfinding algorithms. Etc.
Hopefully not nitpicking too much: it's a win without _many_ tradeoffs. E.g., in the usual places where you'd accidentally get an O(n^2) operation rather than O(n log(n)), the O(n^2) operation is constant-space. In a sufficiently anomalous computing environment with a low enough priority on fast results you might still consciously opt for the O(n^2) solution.
At least in the case of code, this isn't true. The variability comes in terms of change to the system, rather than the running of the system. i.e., if I simplify a process to be less modular and more monolithic, making it more efficient, that also makes it more purpose-built and less flexible. The "risk" increases of running up against a change that needs to be made but is intractably onerous. There's always a tradeoff.
> Fisher's Fundamental Theorem: The better adapted a system is to a particular environment, the less adaptable it is to new environments. -- Gerald Weinberg,"The Psychology of Computer Programming"
It's something everyone should consider in making critical design decisions. Your adaptable, modular system has some risks (particularly in terms of meeting performance targets, increased cost due to increased complexity), but the monolithic system has its own risks (less adaptable to changing requirements, potentially more fragile against attack or damage). Which you choose depends on many variables including your risk profile and anticipated need for change in the future.
If we consider any possible solution, we can obviously imagine adding a completely spurious detail.
A pure trade-off between efficiency and stability would imply that, were I already running the efficient version, we could buy stability by switching to the less efficient code.
* In game theory, "7 hospital beds" weakly dominates "8 hospital beds". But (x') strictly dominates (x, y, z). This is exactly what Pareto Optimality is about. Though perhaps a more colloquial term would be useful here.
Efficiency produces the same output with less input.
Productivity produces more output with the same input.
So efficiency is a measure of input to target output, and productivity is a measure of output to target input.
To make a process more efficient, you figure out how to get to some X output while using as little input as possible.
To make a process more productive, you figure out how given some Y input, you can maximize your output.
So if you fix some goal, say, "we want to be over capacity 1% of the time," then the most efficient way of doing that is probably to have the minimum number of beds that you need according to your predictions about utilization. But you can't really talk about efficiency when you're deciding what your goal is, e.g. whether you're okay being over capacity 10% of the time versus 1% of the time.
For example, how can I make all of today's deliveries with less delivery trucks?
While productivity requires a fixed input goal. Because you want to maximize output while sustaining your desired input goal.
For example, how can I deliver more products per day without increasing the size of my delivery truck fleet?
Often time, improving one can improve the other, but not always. For example, someone could ask, how can we grow profit? Okay, one way is to be more efficient, thus spend less money to make the same revenue. Alright, maybe we use cheaper materials, so now produce the same amount and sell the same, but our margin has increased and we make more revenue. Someone else could say, we need to be more productive. Okay, so you invest in better marketing, and scale production to meet increased demand. You are as efficient as before, but more productive.
So I feel like, reducing the process from X,Y,Z to only X is about productivity. You still have the same number of employees, but since they don't need to waste time doing Y and Z anymore, they can produce more output. That said, you could choose to apply the gains to efficiency as well, for example, hey, because I eliminated Y and Z, I can now cut my workforce in half and deliver the same output.
Having ways to avoid an unanticipated repetition of a process, which would result in bunging up the works for dependent parts of the system, can make the entire flow more efficient. See also 'drum buffer rope' from constraint theory.
optimisation.
For example, if your first statement was 'We only have seven beds because we tightened up our discharge workflow and that's all we need 99% of the time' and your second statement was 'We only have seven admins because we replaced steps X,Y,Z with just step X and that's all we need 99% of the time' they start to line up.
The other one like bad efficiency I would just call "cost cutting measures" not efficiency improvements.
With kaizen you try to accommodate to what you have. So if Bob is slow cost cutting measure would be to fire him. Efficiency improvement way would be observing Bob to see what can be changed so you can get more value without messing him up.
Consider the paradox of finding that a factory crew has no inputs—they are playing cards waiting for an order to come in—and yelling at them to go do other things around the shop like clean and assist other operations, rather than loafing. Or, for another solution to the problem, you might pre-order all the stuff and make sure that the team is always 100% loaded and never has the free capacity to play cards.
At first blush these improve superlativity, no? We are accomplishing everything that card-playing does but we are “faster, more accurate, and cheaper” if we are measuring, say, labor cost per part and the technician time averaged over the parts they worked on. Have we not just found a “novel technique” which is “just a better way of doing things?”
But staring at it for longer you may find yourself less sure. That’s what I mean by complex systems they morph into each other. There are more subtle tradeoffs here. For example when people feel free to loaf when they have no work, you can walk into the shop and ask who’s loafing and why and how you can improve their situation so that they again have proper work to do. There is an increase in latency when that shipment finally comes in and all the workers need to be summoned from across the floor to handle it again. There may be mental fatigue from having to context-switch too much or from having to constantly work on just one thing with no breaks. Or maybe the teams that need whatever they are producing cannot finish their work fast enough, so all of the inventory produced by this team slowly grows until it fills 50% of your factory floor, until you only have a certain amount of space because that’s all you need on 99% of each day.
The point is that the greedy algorithm may fail. In a linear circuit, you short out some resistor with some wire, you know that current is going to move faster afterwards. But in a nonlinear circuit, you no longer know this. In the absolute simplest case, the increase in current rapidly breaks a fuse and everything grinds to a halt. In more complicated cases you have a feedback loop and the increased voltage from the short-circuit feeds back to the earlier stages to throttle the current coming through.
Same with weight loss. People think that they will eat fewer calories and they will therefore lose such-and-so amount of weight. Well, probably. But this is a complex system we are talking about. One of the first things that happens when you start burning the fat is that your body burns your muscle too. This is the same reason that you can't burn fat on your stomach by doing crunches, your system is sending the call out to your entire body that it needs to digest surplus material. The loss in muscle mass appears to be the primary culprit which kicks down your basal metabolic rate and you hit what weight-loss folks call a “wall” where you are literally cold all the time and wearing sweaters and feeling too cranky to exercise and all that, feedback mechanisms which will mean that if you keep eating that restricted amount of calories you won’t be losing any more weight unless you can “break through” it by keeping warm through exercising and thereby increasing your muscle mass back up to where it needs to be and so forth. It’s just that it’s a complex system and the greedy algorithm does not always work for such systems.
In your latter example, it could very well be the case that steps Y and Z had purposes you didn't take into account that makes the new process less efficient in some cases with respect to the target metric.
Either way, overoptimization and focus on specific metrics to the exclusion of others is a real problem. Circumstances change over time and high levels of optimization make processes more brittle and likely to fail when circumstances change.
One makes a tradeoff by reducing overheads and buffers, and the other doesn't have any tradeoffs, it's just a better way of doing things based on novel techniques.