dasil003

This tension is why I refer to myself as a product engineer. Teams where non-technical product managers (ie. your "product team") become feature factories that eventually drown in their own tech debt. Teams where navel-gazing engineering managers (ie. your "engineering team") have too much power end up with premature platformization, and beautiful architecture diagrams while user needs go unmet.

The key is to understand that computers exist to serve human needs, and the less code / UI / hoops to solve a problem the better, and to think end-to-end. If you are able to do that, most systems built by committee have a lot of low-hanging fruit that can be improved without much sacrifice. In practice, political realities and operational concerns may come in the way, but I firmly believe this is the only way to successfully operate as an engineer, and it will only become more important in the age of AI.

Agreed. The law works pretty well for murder/manslaughter because it's a low frequency event that everyone viscerally understands and generally has evolutionarily-ingrained shared values about.

It works less well for other things, like, for example, white collar crime, even when it is low frequency, and even when it is built on financial concepts and norms that have existed at least for decades if not centuries, there's still plenty of gray area for reasonable doubt and spin.

It has pretty much not worked at all for computer and internet regulation except where it could be directly tied back to pre-internet crime definitions (eg. theft of funds). But data privacy, encryption, intellectual property, "hacking", etc, has all been a dumpster fire. And now we pour agentic gasoline on the whole thing? Our legal system is not equipped, least of all not with the current gerontocracy that is beyond clueless about any of this stuff.

This analysis is too one-dimensional, and doesn't factor in social and political factors. As a fellow introverted tech nerd, I understand the perspective, but things don't play out to their logical extremes the way this is projecting. Human factors and social upheaval will come long before any of this becomes a reality as stark as described.

Not that I disagree with the point the author is trying to make vis-a-vis Brooks' Law, but there's another narrative at play here that I find more interesting.

> But someone nudged me: can't you have a look at it?

Clearly this person has some combination of thinking the people already looking don't know what they're doing, and the author being a technical person will be better at it.

From the author's side there is little interest in engaging, perhaps because of a desire not to feed the help vampires, or maybe the meeting is low enough value that an unintentional filibuster is not such a bad thing, or any number of other situational reasons that I can imagine but are not discernible from the context given. Human psychology is fascinating.

It's a strange critique to personify a structural problem in this way. It neither holds any individual accountable for their actions, nor points towards a better structure.

There's a time and a place for both?

Certainly natural language prose can articulate human-level concepts and priorities that code cannot. The fact that LLMs can interpret and apply these things is novel and useful. I agree with the author there is a lot of juice that can be squeezed out of this.

But on the other hand, whenever I start writing code (with or without LLM assistance), there is always a huge amount of learning and iteration that happens. No matter how thoroughly I thought through everything, or how perfect the idea seemed in my head, it always evolved once the implementation was concrete. And furthermore, once you have code running and people depend on it, the code is the reality, and any other documentation or vision statements have a tendency to go stale. Obviously AI makes it easier to keep documentation up-to-date, but the disconnect between the idea of the system and what the system actually does remains.

For this reason I don't see any huge value in deliberately moving to a higher layer. All the same modalities can still be applied with AI: comments, commit messages, PR messages, literate programming, READMEs, AGENT.mds, tech specs, tech 1-pagers, vision statements, etc. I write a doc when it's useful to communicate to one or more parties, otherwise I try to keep everything as minimal and possible, often that means just live in the code.

Yes, let's just get over it.

The system is already broken, so turning over control to AI companies where no human ever sees the patient is a no brainer. There's no tradeoff to consider here, after all the our entire economy is riding on the success of AI, so any concern about human factors that can't be measured by economists is just a distraction from what really matters.

Is it surprising there’s a gap between their vision and reality? Obviously if all that comes to pass they’re hoping the societal implications get figured out to keep the pitchforks at bay.

FWIW no I don’t believe models can replace intellectual work because intelligence is not purpose, and no other entity can tell us what we want. While AI has made building cheap and cutting through the noise even harder, it hasn’t made deciding what to be build any easier, and personally I felt that was already the bottleneck a decade ago (think bullshit jobs, App Store saturation, etc).

Of course that’s just one perspective and I could be totally wrong. But given the situation it makes perfect sense for the big labs to stay focused on building the most valuable shovels in the world and leave the business risk and finding the money for the cash furnaces to others.

Is this always true? I'm honestly not sure. For unit tests I completely agree with you, for integration tests I'm not so sure—validating permutations does not necessarily couple you to code, and may in fact be good discipline for user expectations.

I'm really not worried about this scenario at all. I mean yes, it will be possible for bad programmers to build 173 million line monstrosities held together with duct tape and bailing wire, and certainly these things will exist. And it's also possible that good programmers will be paid by idiots to maintain said monstrosities with no agency to actually fix the root cause. Both of those things will happen.

But a good programmer given agency to solve the higher order problem will always have a path to rewrite, because any "simple CRUD app" exists to serve human purposes, and in order to meet those needs the human interface need to be legible, and to be legible there have to be tight enough inputs and outputs that you can create a parallel implementation without the incidental complexity.

Now you might say this won't be the case for large multi-party distributed systems, and that is true, but that is also true today where all manner of insane cruft and non-sensical logic and workflows exist. But even where things appear totally intractable due to legacy requirements and conflicting logic, there still is a human backstop where if the pain is big enough, any process can be fixed. In practice competing interests may prevent such fixes for a long time, but that is political problem not a technical problem, and it's not inevitable as things can flip very quickly due to the dynamics of human attention and power.

Your first sentence makes sense, but not following your second. Who needs to pick and what are the sides they are choosing between?

They may ask why you don’t have a stamp from the foreign country you were just in though, no? (At least for the ones that still do stamps)

Maybe so, but what I have found in my career long before AI is that a significant of percentage of biz people lack the capability to think holistically or end-to-end. Product people are a little better on average. But at the end of the day it is senior technical folks with product and UX sensibilities who are able to determine what most stakeholders actually want. I find the infantilisation to be going the other way.

Having worked continuously this decade on one massively complex consumer facing app that interacts heavily with real-world scenarios, I can say that the problems of cohesion existed well before the agentic revolution. Tons of features built by myopic teams simply because they couldn't understand enough of the system (both online and offline, meat-space interactions) to make good design decisions. AI actually helps here by being able to perform deeper reading and analysis than any human can do. But then the risk is that it can also produce way more random non-cohesive features.

So far I kind of feel like it nets out to the same bottleneck being good judgment from decision makers. And you might say, no, leadership will just put more and more pressure to add more complexity just because we can and it makes it appear that we are moving fast. But when it comes to real-world apps, there is a backstop which is that things that are too complex and illegible end up not being fit-for-purpose for human problems.

Claude Code and other general purpose AI tools are not like this, because they are literally trying to optimize it for every single kind of knowledge work with zero constraints on use case or target audience other than chasing the next juiciest one they can with essentially unlimited resource. 99% of software is not like this, and when the dust settles I suspect software won't be anywhere near as different as the doomers or the maxxers are predicting right now.

Certainly the level of detachment and alienation has been growing for a long time and is now at quite an extreme. I’m not sure that I would characterize that by saying there is less societal pressure overall though. People feel pressure for lots of reasons and it doesn’t necessarily require the judgment of another individual observing them.

Yeah but Linux has a strong commitment not to break userspace. The average large-scale distributed system made of somewhat arbitrary business logic is nothing like that.

Why would it do anything? Unlike living things that have gone through the rigorous gate of survival-of-the-fittest over millions of years, AI only does what it was trained to do in the sheltered environment in which it has been coddled; teetering atop an impossibly complex supply chain whose only defense against entropy is humanity's greed.

Huge respect for Rich Hickey, and anything he says is worth thinking deeply about.

That said, I think this particular semantic distinction is something we need to think about more from a political perspective than a semantic precision perspective. A tool is something that humans (or animals, or other entities) control for their own purposes. It's important that AI remain a tool of humanity to serve human purposes, following human accountabilities. Because the alternative narrative is that AI is a super-intelligence with it's own agency which we can not control, and while it may make a small handful of individuals incredibly wealthy beyond their wildest dreams, they can't be held accountable because AI has its own agency.

The word "tool" is obviously very old and carries tons of connotation shaped by the world of the past. AI clearly challenges a ton of assumptions about what "tool" and many other words mean. Therefore, while I understand Hickey's point, I think for the sake of our civilization and the moral hazard we are facing, it's better that we consider AI a tool versus whatever narrative the AI cultists are cooking up.

Maybe raw intelligence is not the bottleneck for what can be accomplished with the current data available in a form agents can access it?

Yes, like pretty much everything else, there is gray area. We can have split liabilities for different aspects of these failures. It's long been the case that people may not know what's running on their computers (botnets, etc), and I would argue there should be some form liability for your computer being used to do bad things. It should be proportionate and obviously a lot less than the principals. But keep in mind these rogue AI agents can't just hang out on someone's personal hardware, they need a lot of compute.

The missing piece right now is that our AI safety and cybersecurity controls are simply not up to snuff with the current risks that AI has introduced. Given the increased risk, we need to level up every aspect of our cybersecurity stance. These are hard problems, but not impossible given proper incentives. The issue is that no one is connecting the dots between the creation of these agents and the tools and access they are given, with the harm they may produce. Often the argument jumps straight to "it's going to get too smart for us to control" while glossing over the fact that all the power AI has was directly given to it by human choices. All this cognitive dissonance and willful blindness is caused by unchecked capitalist incentives. Think of the amount of money every single person in the frontier labs stands to make if the most optimistic predictions pan out. This is moral hazard of the highest order, and the only counter-measure that can work in our current social and political system is liability.

I think the capital vs labor structure here obscures the most interesting dynamics going on. I mean, sure there are a bunch of greedy execs that would love to reduce head count and are pretty out of touch so that they have a high degree of not really knowing or caring about the people on the front lines while also not really understanding what they do or if it even can be replaced. This is the same story since the dawn of the industrial revolution, and probably well before that.

But one of the things I find interesting at the ground level is that AI can be incredibly empowering to individuals on the front lines, because suddenly one person can run down code, data, logs, and other tedious scope to cut through layers of bureaucracy that served no real purpose except to manage human bandwidth constraints. Now obviously this is not a panacea, it doesn't work everywhere, can be disruptive, lead to new ambiguities, etc. It's too early to see how it will actually pan out once model capabilities and human ways of working stabilize, and I'm sympathetic to the anxiety this produces. But my point is, bad management notwithstanding, modern AI tends to be most empowering to software engineers and other detail-oriented people, and I don't see this changing any time soon.

Agents run on computers, those computers have owners. We have to maintain accountability and not let people who stand to get rich from privatizing the profits but socializing the risk of rogue agents from continuing to push a narrative of autonomous super intelligence that is “too powerful” to control.

That narrative is bullshit. Intelligence is not power. The power comes from hooking these agents up to everything without proper safety controls. The fact that those safety controls don’t exist yet is not humanity’s problem, it’s the frontier labs problem, and we can’t let them wriggle out of it with an Overton window shift or demanding that the government take responsibility via regulation (of course they should but it doesn’t absolve the labs of responsibility for their own actions).

I agree with your assessment, however I think tech executives are out of touch and lack self-awareness if they think people really want or need this. The whole reason an EA is worth it is because you are making outsized money and your time is worth too much to handle any mundane details that normal people deal with. Along with that there's the whole social signaling of having an EA because it signifies ones wealth and importance.

A free digital EA confers none of that—and it will annoy everyone you know. The only use case I see having legs is dealing with large bureaucracies like canceling subscriptions or medical billing issues, but there I think the common man will lose as the rise of this automation will see a distributed adversarial response from bureaucracies who will happily put up whatever barriers are needed to prevent Muse and other digital EAs from impacting their bottom line.

There's a 50/50 chance I'm just getting old here, but I really think we're going through an epochal shift where new consumer tech won't have the same reception as the smart phone and increasingly addictive bite-sized algorithmic media did over the last couple decades. I think going forward there will be more palpable questioning from younger generations about why do we even want some of these tech products? At some point convenience reaches diminishing returns and we have to think deeper about what we're trying to get out of life.

So what? No one is disagreeing with you about that. The thread was about adding fine grain security controls to traditional OS filesystem access. I was just using iOS as an example, I could just have easily talked about containerization, as a security solution. Of course all those solutions are more limited than a local filesystem—the entire paradigm is designed around full control, you can't bolt on granular security. Heck, even basic UNIX permissions are pretty janky and unmanageable, but at least they more or less work everywhere since they've been around forever. Inventing something now is guaranteed to be annoying and useless.

Sorry I wasn’t suggesting changing a desktop OS operating system. I’m saying that other paradigms offer a more securable substrate.

How would you design such a system to be granular enough to solve for arbitrary application needs without being so complex as to be impossible to tune without false positives and false negatives all over the place?

It's not like people have tried to improve core security models, but in my opinion it's not really compatible with the core filesystem abstraction that programmers and power users of desktop computers demand. I think you need to move to a completely different model—like iOS for example—to meaningfully improve security controls without nerfing the OS.

Yeah. 5-10 years? I estimate I was born 20-25 years before the author (based the about page, not sure how current it is). Yes, there was more opportunity for passionate software geeks because normies were still learning about the internets and the eternal september was just starting to cross the chasm from universities to, you know, the rest of the world. So there was more low-hanging fruit, and the common conception of "success" was much lower ($150k was a massive salary! $10M was a great exit!), but to actually achieve breakout success on the scale that I suspect this person is imagining, always required a level understanding human behavior and ruthless business acumen that most geeks just don't have. I also think AI benefits technically-minded, details-oriented people much more than anyone else. So basically if you would have been a formidable founder born 10 years earlier, I believe you can still be a formidable founder now.

Technically from a legal perspective sure, but there is also the common cultural understanding of these things, and an overton window of how we talk about them. The whole doomer narrative, and pseudo-anthromorphization (xenomorphization?) where we ascribe agency and volition to the algorithm, plays right to the interests of the AI leaders by simultaneously making their products so incredibly powerful and civilization altering that of course the valuations are justified, while also providing a convenient narrative that if anything goes wrong the AI did it and there was no way for them to control this new life form. Privatize the benefit, socialize the risk; this is not a new playbook.

In my experience, AI has given far more leverage to software engineers than anyone else, because they have a feel for what computers can do, what's easy, what's hard, what's performant, or poorly specified, etc. Therefore I think retraining is mostly about being willing to experiment with AI-assisted workflows and not being too tied to old ways of doing things. As long as you are focused on the problems you are solving, I think you'll do okay.

What I'm more concerned about are the younger generation where all the learning tasks have been automated. I have faith that the strong, naturally curious engineering types will still learn to do great engineering, and honestly without some of the mental blocks and calcified assumptions that us greybeards have. But how do they get in the foot in the door and the reps needed to develop senior-level judgment is another question.

In some sense we need the current mania to die down and the long-term maintenance implications of heavy AI assistance to become more apparent. Basically I think we need the AI bubble to pop and then 1-2 years for things to stabilize, and then we'll recalibrate on hiring expectations and best practices for AI usage. The current AI maximalist and doomer views I think are both basically driven by capitalist incentives and wishful thinking from investors and a reality check is going to come in some form (whether technically, socially or politically).