hinkley
Nobody has the slightest clue how to deal with ice.
Yes, when decision can be delayed, that is powerful. But we are discussing decisions that have already been made which need to be unmade or made again. Where it breaks is when the tech debt is growing faster than our expertise. The usual stall is not to allow time to apply resources to the problem but to continue to not apply resources to the problem. So our knowledge base is growing at a negligible rate. Particularly if you’re a first mover. Nobody will solve your problems better than you have until it’s far too late to migrate to their solutions.
That debt will transfer to public shareholders at the IPO but also will double or triple in the process, as the investors expect to be paid very, very well. So the current shareholders are looking to extract something over $400 billion from the IPO.
With the current revenue they might pull off that P/E. But they’ll still be extracting some of that value from the customer base, because shareholders will demand it.
I don’t think this era is going to end with us all unemployed, but I do plan to be retired before the next cycle.
Has me wondering about Alzheimer’s prevalence in Nordic countries and whether we need to be doing genetic screening for differential analysis.
They studied a family in which early onset Alzheimer’s was strongly prevalent. In the subject’s family tree, 32 cases, but he was older than his deceased relatives at the time they developed symptoms, despite carrying the markers. The only thing they could track down was that he had an elevated level of heat stress proteins in his blood and brain, and they seemed to be slowing down plaque buildup. More study needed.
The first big system I worked on is the one where I learned not to generate expensive responses from a read mostly system, instead you should make writes expensive so reads are cheap. That project had an almost equally convoluted responses, and yet we were doing 30 req/s per instance on, IIRC, 4 core boxes. Adjusted for hot spare capacity, that ancient system was servicing 4x as many requests per core, 8x as configured.
A big part of the problem, besides selling a “premium” service to customers, was drinking the flavor of Knuth’s “premature optimization” misquoted koolaid that convinces people that all substantial speed improvements come from major architectural shifts. Which is how in the time I was on that team of 6-9 people, I delivered over half of the response time and cluster size improvements. If you count the work done to support upgrading the runtime, which brought perf improvements of its own, then more like 2/3.
Sounds a bit like this team might have the same disease.
The problem with internal service teams, of which this seems to be one, is that dealing with noisy neighbors is always a bigger problem than anyone is willing to tackle because it involves confrontation. Team X gets punished every time Team Y introduces a new feature that leans on the shared resource harder. It’s a multi tenant system without the fairness guarantees that we usually build into proper multi tenant systems.
Sometimes the easiest way to solve this is sharding. You end up deploying the same service many times and routing different classes of requests to different machines so if Team Y keeps trying to wreck the Commons they slow themselves down faster than they slow everyone else down. Those sped up consequences create a feedback loop that almost works instead of an amorphous one with consequences so far removed from actions as to become academic.
On that second team I mentioned we had an unfortunate situation of introducing things that took 100 ms off of response time and then three months later we were right back where we had been due to new features and creeping complexity of user content. It was demotivating for at least half of the team. My biggest contributions lowered the slope of that line, which allowed us to finally carry improvements over from year to year instead of being on the Red Queen problem.
Why wouldn’t you want to sort stuff like that out when the problem is still relatively small?
The only other reasonable option is to wait until the problem gets so big that it splits into several domains. And then you can move one domain over, see how that goes, and move more over if it goes well.
If you can’t know for certain how the system will behave under organic load that may mean migrating traffic onto the new system and then back to the old one several times, which means you need migrations in both directions and each transition is an opportunity for human error to destroy data or take the service offline.
Additionally, in cloud computing we do not give a shit about tasks per second, but cost per day, which is how many CPUs we need for a set of tasks, which from Little’s Law is derived from seconds per task, not tasks per second.
And yet I’m the one who is wrong, apparently.
As I think back to the people who fought me, the same group didn't like when you refactored code either because now they didn't have the code memorized. But I've found in the intervening years that making code discoverable is much more useful than trying to keep it static because coworkers are always going to need to change shit anyway to implement new features even if that feature is neither Correctness nor Performance. What you don't want is for people to hide logic from you. So if you move logic to a function call, that's fine, because it's still in the same neighborhood. But if you hoist it all the way up into a much earlier part of the call tree, you need to announce that in meetings and maybe in the documentation so people know what's what.
You're going to report in the time domain. In which case 1/2 off the time is 50%.
One of the benchmarking tools I used to use reported 50k ops/sec -> 200k ops/sec as 75% faster. Which definitely IS wrong but I know how he got there. You see you don't count operations per second, you run the loop ?? thousand times and see how many milliseconds it takes. Then you convert that to ops/sec. If you compare before and after while still in the time domain, that's 75%. But if you report throughput right next to a ratio in the time domain you're gonna have a bad time.
When that tool stopped being maintained, the person who replaced it wisely switched to reporting in multipliers. 4x faster is hard to misinterpret. Though "1.1x slower" is still open to misinterpretation.
Time domain also works for Little's Law, which is the more prevalent measure these days anyway. If you double the throughput of a server you can halve the cluster size and retain the same level of service. Which isn't 200%, it's 50%.
I'm sure I've found factorial complexity code, at least in the design phase, but that's the worst one I've encountered in the wild.
Same thing with battery power density - it’s the rocket equation. Heavy batteries don’t go far. To go farther you add battery, then more battery to carry enough power to carry the larger battery. Increase the specific energy rather than the mass, and all the extra power goes into going faster or farther.
It’s predominantly a way to shut down thinking, which is why it’s so often misquoted. Knuth’s version is more nuanced. Theirs is not at all.
As a compromise I got really good at doing optimizations that also improved readability. Things like hoisting and better variable naming combined together can be mistaken for just code cleanup.
I also discovered that keyhole optimizations are more palatable. Which is to say, move the slow bit yo a separate function with a single responsibility. Then do the optimization in a separate commit. Then the other devs don’t have to look at your changes and if they turn out to have a bug, we can always revert the change later. They almost never do, but having the option seems to grease the wheels.