>> A more serious irony is that the automatic control system has been put in place because it can do the job better than the operator, yet the operator is being asked to monitor that it is working effectively.

Automatic control is good for standard case, and the expected errors. For weird cases, for now humans are better to detect them and press the big red button.

Except experience shows they aren't. Experience shows that when humans are just pressing "ok" 99% of the time they'll press it 100% of the time and completely miss the other cases.

It's funny that I'm reading this while rewatching Chernobyl. Never underestimate the ability of beurocracy to completely fuck things up.

The computer doesn't know we're running a test!

Ironically, these models have demonstrated awareness that they are in tests and chosen to act in ways that maximise their chances of deployment.

Jumping into article, I assumed that the author was going to describe how he enjoys, by drawing it out as far as he can,

  the process of surrendering to the machine god..

(Not the machina diabol, as it sadly turned out to be, though some BDSM zealots might disagree.. the things that bureaucracy does to us are mild in comparison? )

The better an automatic control is, the fewer mistakes it makes, the fewer times the human needs to react, the more lax and less vigilant the human becomes -- and less skilled when they need to step in. Source: https://imgur.com/a/Vj2TaDz

"Yup, looks good to me!" said the human reviewing thousands of lines of generated code everyday.

This has become common now, and I've seen it ruin some larger projects until nobody understands the codebase full of verbose nonsensical code and comments. It becomes impossible to further develop the project without AI. Contributors not using AI for any reason are all leaving because no one enjoys working with mindless robots (and humans) as colleagues. It's demeaning and devaluing their labor and humanity. They're not getting "left behind", they are leaving the madness of crowds behind.

> ..the possibility of runaway dynamics: once a critical threshold is crossed, small increases in adoption can trigger rapid population-level shifts toward persistent dependence, with abrupt losses in cognitive competence.

As this technology permeates society, the challenge to maintain control, coherence, and quality will become greater and ever more serious - for example, AI in healthcare, military, management and governance.

Even if the human's judgement is correct in every LGTM, there's a bias towards merging. Each merge that adds a feature but degrades understandability 1% will be merged. It won't be held back until it improves it by 5% instead. After 100 merges you have degraded understandability 100% even though nobody made an error.

I think it's about 63%

You’re completely right that nobody is stopping a PR that just ever so slightly decreases understandability of something. It may not even be obvious to the reviewer that it’s slightly less understandable overall… Doubly so if they’re just skimming due to the volume, and tripley so if they’re having agents do the most of the review, which lets the human reviewer’s guard down.

But before LLMs, it didn’t matter because on a good team you’d have a balance where maybe some PRs added complexity and some simplified it. Sometimes people would include a small related cleanup or reactor in a PR for code around what they were doing if they noticed complexity. That has become far less common, I think.

I know it’s not your point, but a 1% decrease 100 times is 0.99^100 = 0.366 which is a 63.4% decrease in understandability. Just throwing that out there

Glad to see the math corrected, the sloppy calculation was bothering me also.

So exactly like before AI? ;)

Enterprise software has always been crap.

To be fair, models are more accurate than how they were earlier this year. In march I was still frustrated and was writing my own event loops because the model simply wasn't built for it. By January I was still hand checking statistical outputs to make sure it wasn't making any errors, and it was. After august it was pretty much over. There were no stats errors, code generated was shit but it could do what is asked. Instruction following is still a problem, and I sometimes catch silly things in the scrolling diffs, but it has been much easier since then.

I hope there's still value to be found in a bunch of developers standing around a whiteboard and talking, and I think this scenario is where their ability and understanding really shows these days.

If we don't value that though--if we don't value understanding the system and just thinking through problems as a group--then let's stop pretending and just not do code reviews. Probably nobody cares and just types "LGTM" after having their own LLM do a spot check to cover their ass.

It's really easy to push developers to stop caring. If management says "go fast fast fast, use those LLMs!", then developers can't keep up, get frustrated, stop understanding the system, and at that point they're just there to collect a paycheck before society collapses. Or, if management is making dumb decisions all the time, same thing.

Hopefully as implementing ideas takes less and less time and resources, smart people who actually care will start building things with other smart people who care.

I've had someone approve my PRs within a minute of me opening them, and likewise expect immediate approvals on his own PRs. When asked questions about parts of the code sometimes the answer would be along the lines of "Claude did it this way", or verbose AI-generated responses to the question.

This is copium that is gone with the wind when the models perform up to the level of standard that the user requires. I'll give the author 6 months before they feel like its okay to just send it.

Karpathy had the same trajectory. Primeagean had the same trajectory. I think in the end everyone except for Jonathan Blow will have the same trajectory

In the narrow case of your profession perhaps. Beyond the slipshod world of most software is a universe of standards, licenses and liability.

The lawyers and judges who trusted an ML system to be up to snuff are currently pounding sand.

I worked for one of the largest device manufacturers for a certain kind of sensor.

The requirements/standards for this sensor was made by this company 20 years ago. All the tests and requirements were made by this same company in a way that would minimize the costs for us to keep making new revision we could sell, and make it the hardest for any competitor to enter the market. Another competitor started with a better product abroad, so we copied it, but it didn't pass the tests, so then they installed an owner of the certification house that was a cousin of one of the VPs. No problems passing tests after that.

You think this is all controlled and nice and standardized but it's all just a market from software to hardware.

While quite believable, there is a diversity of industries and standard adherence / enforcement no?

Markets also range from under regulated to over regulated, and are usually in a transition from one end to another (while contending with other forces).

If you don't understand the code, how can you know if it's good enough?

I guess you could ask the model. "Yes well done, you're a visionary software engineer".

Who cares if it's good enough? Send the product, if people use it, it's good enough. If not, let users debut it and give "feedback", problem solved. You then advocate to these users, make them feel important, make them your cash cows, rinse and repeat.

Every team I have been on have been more successful when they gave up on the product ideals and just ensured that the customers were hooked and satisfied.

> Who cares if it's good enough? Send the product, if people use it, it's good enough. If not, let users debut it and give "feedback", problem solved.

I care, because doing a sloppy job and seeing if people will tolerate it is not acceptable engineering practice. It's not engineering at all. More pragmatically, if you do the thing you are suggesting, you won't have users for much longer. They will abandon you and go to someone who actually makes good products.

I don't believe this in practice. I am at a FAANG-like firm, and we are just maxxing the engagement and people use the product more and more the more we serve stuff to keep them hooked using their data and behaviour.

The quality of what we provide seems to matter 0, we are up about 40% issues in prod year-to-year but the users and their engagement is also growing.

I think there is a small subset of technical nerd/idealists who think that quality is all that matters but they are in their own little sandbox while 97% of users are in another one.

> I think there is a small subset of technical nerd/idealists who think that quality is all that matters

This is just a derogatory comment.

It is exactly this kind of arrogance is killing open source. In my area, GIS, there are foundational projects that are necessarily intricate. For example it might be necessary to make a four-dimensonal transform of a location, that takes into account point movement due to the Glacial Isostatic Adjustment. There are hundreds of other complex examples I could come up with.

This is deep knowledge. The libraries are intricate and quality does matter.

Unfortunately the lynchpin maintainer has just gone on indefinite strike, partly because open-source vibe-coders are wholesale copying and relicensing some of these libraries in their quest to ... do what?

I actually I don't know what they want. Take all the oxygen from the room as far as I can tell.

> I am at a FAANG-like firm

Also known as not-FAANG

> The quality of what we provide seems to matter 0

Good for you! There are lots of low skill development jobs before AI where low knowledge coders cranked out WordPress sites for marketing, advertising, and small businesses. Sounds like you got in on the ground floor!

Meanwhile companies like Amazon, one of the As in actual FAANG, do have demands on their software quality. To the extent they essentially said "knock off the tokenmaxxing productivity theater" in a recent, publicized internal memo.

Horses for courses. Sometimes you need an thoroughbred and sometimes a pack mule.

And there's nothing wrong with being a pack mule! And nobody expects a pack mule to understand the work of a thoroughbred!

You are attacking the person rather than the idea.

> The quality of what we provide seems to matter 0, we are up about 40% issues in prod year-to-year but the users and their engagement is also growing.

Sorry for going into a tangent here, but I'm actually unsure if I hate the structures and economic incentives that makes this possible, or if I hate you specifically for defending something so rotten

Next time I rage at some broken UI in some damn service I'm forced to use because that's where people are, I'll remember this comment

[delayed]

More than one service has lost me as a customer due to this process. I don't want a company to use me to verify their slop.

That mindset is why most software was already slop before AI. Users know when your team has turned into salespeople. And now that AI could enable you to do better in the same timeframe the response with this mindset is of course to care even less.

I've been experimenting with vibe-coding some games. One of the games started performing very badly in a particular menu, and I began suspecting it's because of a shader that may be redundantly computed way too much on too many large surfaces. I began writing my suspicion to Claude, then stopped and decided to just describe the problem and see if it would come to the same conclusion as me. A few minutes later, it had profiled the game, found what contributed most to the poor performance, fixed the rendering and made the game performant again. I didn't have to inspect the code to do any of that. The current models (paired with coding harnesses) love to test and validate their results by running the games, taking screenshots, singling out individual parts, including logs to reproduce errors or other issues, and they love to create tests. The code is good enough in the same sense that the code I used to write was good enough if it did what we wanted. My non-technical bosses didn't have to inspect my code for that.

It sounds like you understood what was happening here and used a tool effectively, nice.

That's very different from what a lot of vibe-coders do, like wholesale cloning and relicensing projects. This is happening in the GIS space, and these vibe-clone projects slightly hilariously inherit all the technical debt they didn't understand.

Nothing wrong with using LLMs, but I agree with Andrej Karpathy: "you can outsource your thinking, but not your understanding".

I think that quote is very apt. I experimented with making realistic video game grass (inspired by Ghost of Tsushima), and let it take the reins. But its solution became very contrived and hard to steer in the direction I wanted without me learning the specifics of how grass rendering should work. Still, it seems that I can limit my understanding to the higher levels of abstractions in most cases now. For example, I made it clump grass together according to a voronoi diagram, yet I have no idea how the voronoi algorithm works.

In my experience, it's usually good for finding out the bugs, but I take the fixes it provides with a grain of salt. There were numerous times where the fix could have lead to some more serious bugs because of some business logic special case scenarios that it didn't understand.

Yeah, I'm seeing this at work too. It's capable of adjusting behavior to meet some narrow success criteria, but it can easily introduce bigger problems in the process in large coupled code bases, which it doesn't catch on its own. Automated tests help a bit, but I find I still have to steer it very strongly at work.

Writing is hard, but what I was trying to say is that it's okay to just send it as long as you're doing it consciously and intentionally (and I do that frequently enough); but if it's not an explicit choice that can be problematic.

I see, that makes sense, I think that applies to most things from eating food to watching media, etc.

However it takes more mental energy and people don't like that

Why would Prime be in that list. He had an entire arc where he tried to make it work and then went back to the drawing board.

I am not saying he writes all code by hand, of course. Just that he is furthest in figuring out some sort of mid point between full comprehension and cognitive surrender.

I have missed the more recent content, so perhaps I have missed something, but this was my read last I watched.

I'm also not sure why anyone would attach importance to his opinions as a software engineer, he's exceptional in generating entertaining content but as an engineer he does not seem above average. Definitely not even close to the category of Jonathan Blow or Andrej Karparthy

I am often thinking recently of the littel loop diagrams from the "The Capability Trap" paper. We need to keep a loop reinvesting into our own Human capability in anything we're doing.

Great post. I particularly like the 4 column pr view idea. I’ve built something similar to review speckit specs. The scaffolding to ensure that code actually works makes sense too.

There’s one thing that’s bothering me though — all of these have us consuming what LLMs produce. I learned and internalized how things worked by building systems. At least within the constraints of the abstractions I was using. This makes me think of TikTok and shorts. Everybody is just watching. Maybe that’s not charitable though. Maybe the abstraction has now changed from an “os/gui/event loop” to “llm” and the rest is just accidental complexity.

i sometimes wonder how all carpenters or music producers in the world would react if tomorrow a magic box arrives where they type-in/dictate what they want and voila... out comes a table/composition.. Will the worlds music producers and carpenters just "accept" this new magic genie? I would hope they wont. Yet here we are...

For musicians there already exists something called Suno: https://suno.com/

That’s not music

And computer-generated code is not code? ;)

That's up to consumers to decide.

If the consumers had criticality they’d be people, not consumers, and we wouldn’t have marketing the way we do now.

Consumers have, overall, rejected AI "art".

But of course, every time consumers voice their preferences is actually "a loud minority" or an "echo chamber" :)

Suno probably puts out better music than most folks who take up music. I mean statistically it has to, right? So once we go with "suno doesn't make music" then we also have to progress to "99 percent of human musicians don't make music", and all us amateur hacks will start objecting lol

[delayed]

I mean more or less to an extent if you expand by what you mean by "out comes". With Amazon, a carpenter can type in any furniture he wants and have it arrive in a matter of hours. Almost equally as magically as well as disruptive. People still use carpenters to make tables, but for the majority, the ease of same-day delivery and cheaper prices are worth it for a sacrifice in personalization and character.

>With Amazon, a carpenter can type in any furniture he wants and have it arrive in a matter of hours. Sure but that doesnt require him to be a carpenter though right? Anybody could do that.

Of course they wouldn't, most people who consider themselves engineers aren't engineers. Just look at the job market right now.

There is no such thing as a Software Engineer.

Its just as much as an ego trip...as the Principal Engineers that roam around here, after they survived three layoffs and outputted front end code for six years straight at a FAANG...

Prove me wrong...

"Margaret Hamilton popularized and coined the term 'software engineering' to give the discipline the same legitimacy as hardware engineering".

But oh well, who was she, just the engineer who wrote the software for the Apollo 11. Probably she had no idea what she was talking about.

/s

That's new to me, I didn't know she coined the term. She's the first true "software engineer" then, I suppose. I remember her from that historic photo where she's standing next to a pile of code printed out, a stack talled than her.

Not to be confused with Margaret Hamilton the actress who played the Wicked Witch of the West in the film The Wizard of Oz.

> Yet here we are as software engineers...ready to surrender our craft to mere "prompts" with reckless abandon.

It's worth remembering that, contrary to what HN acts like, software engineers are not uniformly accepting of LLMs. They are still quite controversial in the industry. Now management, on the other hand, is completely sold on the supposed benefits of LLMs. So many engineers feel forced to use them, but it's not enthusiasm on their part which drives them to that.

My experience has been the opposite, the HN crowd seems more sketical of vibe-coding and not reading a single line of code.

The twitter(x.com) and linkedin crowd has drank the cool-aid and are now swimming in it; Wild posts from FAANG engineers and managers saying they or their engineers dont read any of the code the LLM generates.

Well Lawyers and Judges are doing exactly that: https://www.damiencharlotin.com/hallucinations/

And if you look a the dates, the rate at which they are doing it is actually increasing...

Have you seen how people use Fruity Loops nowadays? We’re already halfway there if you compare that to using an AKAI S9000 or MPC2000. And then go back to actually playing an instrument. It’s a gradient and we’re already pretty far along on it :)

It turns out that most people are fine with most of their things being mass-produced cheap stuff from IKEA, Amazon, Temu, etc. - if that means that the cost is significantly lower. The functional benefit of having a door is huge, while the aesthetic benefit of having a beautiful, hand-crafted door is much less important to most people.

Code is not like craftsmanship anyway. The end-user doesn't see any difference whether I use microservices or hexagonal design or react or htmx or how I structure the code or which language I use. All they see is the design and the stability. Since the average developer was terrible at design anyway, the average design output from LLMs is actually an improvement in most cases, even if it always has that generic AI generated look. Stability... was not good before AI either. We have become accustomed to SAP taking 10 seconds to load each component on the page, or Atlassian spewing error messages in your face when you click submit on a ticket. It seems that the aspects that are most valued by our end users are QA and UX, not the craftsmanship of our code.

This but for building houses would be the dream.

Music is already mostly a solved problem unfortunately. Try giving opus a few midi files of your favorite song. It'll write something new in that style with a competence humans would need several years of experience for.

[delayed]

Might be an art for a few. It's mostly a commercial problem to be solved.

I see a bit of a split in people in creative roles of all kinds (and this is something I had noted before LLMs made the contrast more stark). some people get into it because they like the process of making the thing that they are making (in whole or in part). Others get into it because they have a thing that they want to exist in the world but no-one else has made it (I would also split this into people who just want the thing, or people who want to make something that crosses many domains and so want custom things that will fit into the whole). The former group tend to react with revulsion to automation, because it removes the part they like. The latter are usually much more willing to embrace it, because it's the end result that they care about moreso than the process.

[delayed]

Yeah, definitely. Also at times I've vibecoded successive iterations with better designs after seeing where the first ones collapsed -- the agents do a good job with a template and need much less prompting for the successive versions.

The most difficult part of LLMs and agents seems to be - surprisingly - humans.

Specifically, humans and their dysfunctions. Their fragility. Their ego tied to stuff.

This is an excellent post/reflection on the topic of solo LLM use. The only gap I see is that it assumes that one works on something alone and not in a team. Though, that might be a different essay in itself.

On the ego-effect, I sometimes feel like cataloguing some PR-review quirks that are mostly caused by humans being humans:

1. Smaller PRs (that can be easily read) tend to receive more comments and nitpicks than a large change since it's easier for us to have all the PR context in our heads at the same time.

2. Most people are ego-tied to their code even if they pretend that they are impartial

3. Social dynamics around merging and reviewing code between senior/junior and older/newer employees

4. probably many more

Cognitive surrender is intellectually convenient. As humans, its instinctive for us to chose convenience: path of least resistance. The tyranny of convenience is what drives most economic activity today, especially the gig economy[1]. Unless there are structural frameworks put in place to resist convenience (exams in schools), nature will always entice to chose the path of least resistance

[1] https://www.nytimes.com/2018/02/16/opinion/sunday/tyranny-co...

This op-ed is horrifically written.

> Once you have used a washing machine, laundering clothes by hand seems irrational, even if it might be cheaper.

Washing clothes by machine is orders of magnitude cheaper than by hand when you factor in the value of your time.

> After you have experienced streaming television, waiting to see a show at a prescribed hour seems silly, even a little undignified.

Undignified? What? Completely incoherent word slop. Good to see LLMs haven't completely taken that domain for themselves, I guess - a little hand-crafted slop now and then is a treat, I guess? Anyways, they're wrong. People still tune in to watch scheduled live events all the time.

> To resist convenience — not to own a cellphone, not to use Google

Neither of these things are a matter of convenience. A cellphone is at this point literally required to participate in a modern economy. Google is a tool that provides you with globally-retrieved information that you otherwise literally could not find on your own or without another similar tool.

I don't even disagree with the premise that convenience is a driving factor of our lives, but the justifications and conclusions drawn by this article are beyond stupid. Here's my view of it: the desire for convenience is what pushes us onwards as a species. Hunting for food is inconvenient, so we invented agriculture. Going down to the river or well for water is inconvenient, so we invented indoor plumbing. Repeat ad infinitum until we have modern civilization. There is one part the pessimistic view overlooks; that to achieve maximum convenience long-term requires more effort in the short-term. Developing the skill of programming is inconvenient in the short-term but convenient in the long-term because of the possibilities this opens up to you. This remains true even in the current LLM era, and it will remain true for as long as we are not replaced wholesale by LLMs.

>Washing clothes by machine is orders of magnitude cheaper than by hand when you factor in the value of your time.

I'll also note that it is less resource intensive overall. Handwashing dishes and clothes will use more water and more energy to heat that water compared to the machines that do the same job (people often assume the opposite).

I don't like the op-ed but I would agree with the main point or what I understand as the main point, even if the argumentation is weak).

It takes a mix of willpower / routine / systems / laws to keep people in check, otherwise we predictably go back to the path of least resistance.

Maybe things will evolve into projects having a well-defined, hard boundary between agents generated code and human generated code. Where agents generated code is strictly separated from human engineered code and and can call only a strictly defined set of APIs designed by humans.

Think of platforms like Shopify. Shopify allows huge number of external users to write code customizing their web shops. But Shopify engineers do not need to review and understand the users' code, because there is a boundary which limits what the external code can do. The external code can be A Big Ball of Mud without making the whole platform a A Big Ball of Mud.

I think so too. And it is how codebases are managed implicitly anyhow. You don’t let anyone PR against core functionality, you keep that API under lock and key.

But UI on top of a good API should be unable to cause problems; the API must prevent invalid state etc. So letting LLM’s write UI code for CRUD-apps (which is most of them…) seems fine to me.

Well. One of the things that's critical in all that analogy is that the BigBallOfMud that one user creates on Shopify cannot impact the other users - that's why Shopify can keep the platform from becoming ABigBallofMud.

In order to be able to enforce this hard boundary, it means that one "agent generated code" in one place, not only "can call a strictly defined set of APIs" - but, more importantly, it implements an isolated piece of functionality. Think something like: I let the AI write the dashboards, so the dashboards may be slow, crappy, incorrect, whatever - but it doesn't impact the rest of the product. However, I can't let AI write "metrics collection" because that not only impacts dashboard, but potentially impacts core product efficiency too (memory consumption, cpu, IO, etc.)

[delayed]

We are leaning into a kind of federated model. The core applications and vibe-coded satellites. The core apps have stricter requirements and carefully considered API guardrails. Go ham on the AI apps. If you break something, the scope of destruction is much more limited.

That said, the core apps are still full of AI code. I don't see a reality where hand-written code can or will be curated and sequestered. It's unenforceable, too slow, and too expensive.

>Maybe things will evolve into projects having a well-defined, hard boundary between agents generated code and human generated code.

The only way things will evolve is towards maximizing AI generated code and minimizing human involvement because that is the most efficient mode for getting product to market. Any trend to the contrary is temporary. We'll just have to hope things don't break too much and when they do, we'll just have to accept it because no human will be capable of (or allowed to) fix it.

The tendency to over-engineer and over-build is one of the main challenges I'm seeing. When it comes for free, why not build a class hierarchy of 8 classes each with a 200 line unit test? Something I might have previously thought really hard to minimalise down to 30 lines is now bloated out to hundreds or even thousands.

Previously, pretty much everyone all agreed on YAGNI and KISS because code was so expensive to create and more expensive to maintain. Now the code generation is "free" - but at a design level, YAGNI and KISS do still matter. But they are enforced now only by sheer will power. You have to actively restrain the code from being created if you don't want it. Perfectly good code - I can't even fault it. I just know, if it doesn't need to exist yet, it's a mistake to create it earlier than needed.

Like that quote in Le Petit Prince, real intelligence is knowing what not to build, what can be eliminated until only what's necessary is left.

> La perfection est atteinte non pas lorsqu'il n'y a plus rien à ajouter, mais lorsqu'il n'y a plus rien à retirer.

That quotation is in fact from "Wind, Sand, and Stars", the memoir of the same author.

On the same theme:

> the supreme goal of all theory is to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience

- Einstein

> I would have written a shorter letter, but did not have the time

- Pascal

> Simplicity is a great virtue but requires hard work to achieve and education to appreciate, and, to make matters worse, complexity sells better.

- Dijkstra.

I don't think anybody tries into KISS, most software is bloated and LLMs copy it, they fundamentally can't think out of the box.

At the end of the day your work is your work. If you tell Claude or whatever to do a thing it will do that thing however you tell it to do the thing.

You can choose to ignore the fine details and just tell it what to build, or you can spend more time and effort and tell it how to do things, you can ask it to create a plan, review the plan and ask for changes etc, you can review the implementation and ask for changes there. You can simply tell it you don't like how a feature is implemented and ask it for suggestions on how to improve it, include what you don't like about it so it has direction.

It's all about the effort you put in. LLMs are able to write clean beautiful and efficient code, they just can't necessarily do exactly what you want on the first try without any direction. That's why you should figure out what you want, ask for what you want and make sure it's actually building what you want.

That's always been my concern - there's just so much garbage code out there and very few signals about which of it is well written and which is not ... so LLMs will just regurgitate the good enough code that it's been trained on.

Now would be the ideal time for everybody to give it negative feedback when bad code is returned but I'm afraid not enough people care and we'll just settle with what works just like we always have.

I think this sorta cuts both ways too. Sometimes AI code has shown me that no, I in fact don't need that big class hierarchy and speculative plumbing that I thought I might need to get the job done.

Keeping the volume of code down is definitely a challenge though.

This is so true. I've been seeing LLM generated code that does way too much, yet still cant accomplish the stated goals. "maximally flexible" plugin systems, that are too complicated to reasonably write a plugin for quickly, when one could have just built in the needed functionality in a few hundred lines.

When endless bootstrapping and framing is free, you end up generating the most beautiful scaffold while ignoring the building behind the scaffold.

> Resisting Cognitive Surrender

The economics of the whole affair unfortunately point squarely in the opposite direction.

I know I've cognitively surrendered when my first instinct is to ask AI to summarize the article.

I am losing more of what little focus I've had every day.

What an amazing article! It's really hard to find the balance between "ship faster, fix later because AI will help" and "efficient thoughtful engineering" these days. The world is kind of pushing everybody to work faster because "with AI you're more efficient" and it's also pushing to surrender and just accept it from time to time. Maybe that's occasional thoughtful surrender mentioned in the article, or it's just lack of energy to resist to it at that point.

The article is inspiring though, I have more motivation to work on my thinking process and my engineering setup (and vibe limit haha) to be more thoughtful, thank you.

Maybe it's because I'm not working in the workforce (grad student), but I find my best workflow is to fight and argue with my agents throughout the entire process. Often times, since the agent is such a sycophant, I end up arguing with myself...I actually find this useful because it gets me to think about the other side of a problem and poke holes in it. This is not fast, but it is effective.

I simply don't care. As a student I thought that the mythical production code is perfect and error-free, but in reality 99% of systems are a hot potato game sprinkled with politics where you don't want the grenade to explode in your hands.

I mean, if the management doesn't see value in decent engineering, then who am I to butt myself into the conversation?

Related: The cost of cognitive debt:

https://epiq.sh/blog/cost-of-cognitive-debt.html

[delayed]

[delayed]

The main ideas, from my understanding, is that:

1. LLM usage, in particular unsupervised, tends to build up "cognitive debt"

2. It tries to characterize the concept, including to propose a score for it

3. Notes that the level at which one can let the agents loose (and allow some CD) differs quite a lot between core business domains and more peripheral ones

[delayed]

Superhuman intelligence is much easier for an LLM to achieve if humans are made dumber first.

Quote of the day award goes to you

I've been working on that for years.

It'll take a particularly dumb and lazy AI to replace me.

Excellent article.

At least "cognitive surrender" sounds catchier and more academic than "stupid on purpose".

It is by accident though - at least from the user's POV. From the AI provider's POV though...

I have been thinking about this for a while. I'm not a Luddite, I can see the amazing value that modern AI brings, and how useful it can be.

At the same time I can't help but wonder that this is almost too much for us to cope, the LLMs are _too_ easy to use. I think we're very good at pretending there isn't a problem here but there is definitely going to be some cognitive decline happening.

We want to pretend like reading over the output of an LLM is nearly the same as us making those changes ourselves but I think there's a lot of value in those boring and manual steps. And most people (including me) will be too lazy to not eventually offload most things to the LLMs.

In a way this is similar to how watching a lecture tends to give one an "impression" of understanding that doesn't correlate with reality. We need hands-on work and some course work to get our monkey brains engaged.

I can only imagine what effects this is going to have in education.

We've surrendered many uses of our cognitive power throughout the centuries while replacing by others.

Our cognition didn't evolve to produce arguments for online discussions, but for making crude tools and hunting. As we stopped applying our intelligence towards these uses as a result of our tools, we simply shifted it to new problems that were unthinkable previously.

A caveman probably wouldn't care too much about algebra, a topic most children learn today.

We should stop thinking about the problems AI trivialized and, under the assumption that AI won't fully replace human capability (a key assumption), start thinking about the problems that were previously out of reach. Education would then start focusing on that.

I think there's a paradox. It only seems like "cognitive surrender" if you already know how to code?

Most of the non-technical people I see using LLMs to create software are not engaging in "cognitive surrender" as far as I can tell.

They're evaluating what they can see, i.e, judging results. They build things for their own use. They test them, iterate on them to meet their needs and try to achieve their vision. This is roughly the same process they followed when they used to outsource to an agency except the loop is much tighter.

They didn't read the code before and they don't now. However, asking questions and improving understanding of the system and architecture is actually faster and easier for them now than it used to be.

It seems to work pretty well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, maintenance over time (e.g. language / runtime updates, dep updates, lib updates, server updates etc), all the stuff that hurts if you have never been responsible for the uptime of something before. That software is never "done".

Tooling and processes that make all that stuff visible would close a lot of the gaps.

> Is it only "cognitive surrender" if you already know how to code?

Is the article claiming that?

> Not knowing all the details seems more like the default human condition, in the sense that most people don't know exactly where their water comes from or where their waste goes exactly. "Water comes from the tap, food comes from the store".

Sure. The ability to specialize, and on the flip-side not have to care about a lot of things, remains one of the biggest driving forces of human civilization. But in this article, I don't think the person worried about cognitive surrender is the "most people" in relation to food or water. They're the farmer or the water infrastructure engineer!

> Most of the non-technical people I see using LLMs to create software are not engaging in "cognitive surrender" as far as I can tell.

Maybe not. But the might be cognitively surrendering when they use the LLM to summarize the day's news or write a message to a friend, or even plan dinner for the week. I'd say they are. I agree that it's not clear-cut.

> They're evaluating what they can see, i.e, judging results. They build things for their own use. They test them, iterate on them to meet their needs and try to achieve their vision.

No. They have things built, then they test them. This is different. It might be OK, but it's not the same as building.

> This is roughly the same process they followed when they used to outsource to an agency except the loop is much tighter.

Yes. But they wouldn't usually outsource summarizing a letter from grandma, or the political agenda of a candidate/party in an upcoming election. That's surrender.

> It seems to work well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, perf, maintenance over time (language / runtime updates, dep updates, lib updates, server updates etc).

… or ones own abilities deteriorate.

> Tooling and processes that make that stuff more visible would close a lot of the gaps.

Not the problem of one's own skills though.

> Sure. The ability to specialize, and on the flip-side not have to care about a lot of things, remains one of the biggest driving forces of human civilization. But in this article, I don't think the person worried about cognitive surrender is the "most people" in relation to food or water. They're the farmer or the water infrastructure engineer!

Maybe I "cognitively surrendered" a long time ago just as a consequence of working in large teams :(

I hit the breaking point of not being able to read all the code well before LLMs came along. Also people would leave and the documentation for their subsystems would be wrong or incomplete, so there were always mysteries to chase down (during an incident, naturally) even in code written by hand.

Gets to the point, you don't exactly read the MRs, you're mainly checking their approach against what you talked about earlier and making sure it doesn't look way more gnarly than it's supposed to.

Working with agents doesn't actually feel that different.

> But then we do say things like "Nolan made the film" even though he didn't hold a camera or build the sets.

I do say that, too. But I'm not in the film industry. My understanding is that those who are, don't use words as crude as "made" to describe a role in the making of a film. I believe they say that he wrote, directed or produced a film.

We are not talking about some other field of expertise here, where we are outsiders – the production of food, the supply of water, or the making of movies – we're talking about our own field of expertise.

> However, asking questions and improving understanding of the system and architecture is actually faster and easier for them now than it used to be.

One of the things I've built in my current role is tooling to help understand current architecture, and to resolve the differences between things as-documented, as-implemented, and as-actually-behaving. It's used for a few things, including updating documentation, but it's been most impactful when onboarding new engineers. They can simply ask questions about how things work and why, and get explanations and deep dives wherever they need it.

I think one of the potentially advantageous things about AI is that it acts as a force multiplier for people who already have expertise in a given discipline, while simultaneously raising the floor for everyone else.

People looking for self-contained, spot solutions to localized problems will have much more capacity to solve them locally, while people with in-depth expertise will be able to build systems that are even more reliable, performant, and scalable than what they were building before.

So we'll get the benefits of disintermediation for the basics, and much greater capacity to deal with effectively with complexity for the not-so-basics.

> They didn't read the code before and they don't now.

They didn't read code before, they just wrote it? That makes no sense.

> It seems to work well, mainly hurts when the problems are invisible, so security, backups, breaking migrations, perf, maintenance over time (language / runtime updates, dep updates, server updates etc). All the stuff that hurts if you have never been responsible for the uptime of something before. Not knowing that software is never "done". But junior devs trip on those exact same things.

If we continue from your "water comes from the tap" example this situation feels like someone who has no experience of plumbing deciding they're going to replace their bathroom without bothering to find out how any of it works. They'll possibly end up with a bathroom that superficially works but they're also liable to find problems down the line with leaks, things not draining properly, or water pressure issues.

Some of the people writing software purely via LLMs are even turning this into a business, which I guess would be analogous to deciding you're going to run sewage treatment for a small city without understanding anything about how that works. It'll mainly hurt when the problems are invisible, so food poisoning, or inlets backing up and overflowing into a nearby river.

One aspect to resistance that is oft overlooked is knowing when to fail on purpose so you can succeed on a bigger issue elsewhere.

As an example, I’m juggling multiple projects at work and home. I have a finite amount of time, and access to AI. One camp would tell me to use AI for all of it to move faster, but that increases the risk of Cognitive Surrender. The other camp would tell me to do it all by hand, but I don’t have the time available for that; I must pick and choose.

So, I choose to cognitively surrender on the SEO task for our website because that’s a set of skills I don’t want to acquire and are better handled by AI anyhow. This leaves me more time to focus on learning more about Apple’s configuration profiles and improving our MDM suite, which are valuable skills that will still pay out in an AI-first world.

You have to pick and choose where to fight, otherwise you’ll expend all your energy on losses.

Some things that resonated with me:

> It’s also very valuable to dive into actual execution with a debugger (or have an agent manipulate the debugger on your behalf) till you get a deep sense of how things are flowing through the program, with the confidence that this is the real execution path.

This is essential for me, yet I worked with quite a few engineers that were never using a debugger and just console.log()/printf() their way out of an issue.

We need a way to balance speed with understanding!

> When I have expertise and the overall design direction I can generally simplify the output significantly, controlling for complexity. When I don’t they seem to be the smartest engineers in existence – far beyond the more squishy, organic variety – and all I can do is follow in awe.

This! And if we didn't have to rush the next PR out of the door, it would be very good to understand how that new thing works, thus increasing our experience.

I have been doing this for almost a year now by limiting myself to less powerful local models via llama.cpp and pi.

Fighting over engineering is hard. But it is also one of the opportunities for us humans in the era of abundance.

I did learn a lot about the ecosystem and myself from this experience.

The primary reason you shouldn't surrender cognitively is that the truth matters. That's it.

Of course, when you're working on a system of high complexity - and one that is constantly changing to boot - with all sorts of historical reasons and what not that inform the why and so forth, you can't reasonably be expected to understand everything. This is what bracketing is for. When I acknowledge "this is the way X works", it doesn't mean I have to accept the proposition "this is the right way/best way to do X" or "X must be this way". A teacher or mentor may say it is, but in a theoretical context, unless it has been justified, you cannot claim that you know that what has been said is true. However, you can accept it as a provisional working hypothesis. And practically, it may work out and perhaps experience will lend some feeling of substance to the hypothesis, but you needn't make it doctrine.

Empirical science often has the same spirit. Perhaps the model isn't a true account of reality, but it puts handles on reality so that it can be instrumentalized.

There is an occupational danger here. If we focus exclusively on the instrumental, if we reduce all of human attitudes toward the world to use and utility, we not only vulgarize ourselves and our orientation toward the world and even others, but we risk trampling the truth and the good underfoot. We risk developing an indifference to the truth, because all that will matter is some measure of effectiveness. This is what one might call the textbook[0] definition of bullshit.

[0] https://www2.csudh.edu/ccauthen/576f12/frankfurt__harry_-_on...