trjordan
I'll go out on a limb and guess that at a hyperscaler, shipping a bug to prod means hitting canary or 1% of traffic. Seems fine!
I think my point still stands, even as the shape of failures shift. There's certain ones that still hurt (data corruption, exploited security holes, etc.) because they're hard to notice. It might even be that the primary goal of ownership is to develop intuition about where those failures are likely to happen.
The failure that actually matters here is product failure. Angry users. Security incidents. Downtime. Broken workflows.
Stressed and uncomfortable developers have, historically, been a leading indicator of product failure. Software is complex and when devs can't handle the breadth of ownership + the depth of expertise at the same time, it means bad work that leads to product failure. It's not clear that agents fail in the same way, or that our collective intuition about how and when to trust agents is sufficiently developed to predict product failure in the same way.
As anecdata: I know a bunch of growth-SaaS/last-batch-of-IPO companies where the devs are all stressed, the feature ship rate is up, and incidents are down. All at the same time.
So yes, quick learners and just-in-time depth is important, but so is intuition about where discomfort is going to produce true failure. The discomfort isn't a feeling to throw away, simply trusting that you'll learn everything in time. Ownership requires holding a model of the system in your head and proactively steering where you dive.
This is just the compute, which primarily covers training + inference. They still have to run the rest of the business. If you think AI should look like SaaS and spend 10% of their revenue on compute, that's $1T in revenue. If you think it's a hardware business, that's 30% margins and perhaps $200b in revenue. I went with something in the middle at $500b.
They won't disappear as a company at $200b in revenue, but I don't think that's their aspiration. These commitments imply to me that they're confident they can do much better.
Anthropic thinks it is. The got the demand / datacenter match right in Q2, got great margins, and committed hard.
They've got:
- $114b committed to Google
- $110b committed to AWS
- $161b committed to Broadcom
- $120b committed to SpaceX
If the party continues, they'll go from $50b in revenue to $500b and be fine.
But if they don't, it's going to hurt. And that's their only choice, because otherwise, they end up with a frontier model and 50% of the capacity and get eviscerated on either availability or last-minute datacenter leases.
It's a big business, but it's not an easy business.
- OpenAI: it’s 9pm you promised this person this note by EOD. Here’s what it should say. You want to draft it or me? (Wow yes I missed that, thank you)
- Claude: hey you logged into Docusign, is that ok?
- Grok: I tried to run a slack search and accidentally sent 3 messages to #general. Just like yesterday.
I presume we're still talking about software. Auditing proves what?
Accountants audit that the books are consistent. It's extremely hard to fake consistency and do fraud, especially when there's 3rd parties involved. The edges of the arithmetic get nailed down, because bank payments were sent and invoices paid. So the internal consistency is checkable and valuable.
But software. Are you proving it does what you want? But you didn't write down what you want. Or maybe you're into spec-driven development, or detailed wikis, or whatnot, so you did. Most people aren't, though, and those docs are already out of date anyway.
I think there's something here, because we're all telling our agents what we want the software to be. So maybe you could audit against that. But this isn't the same as accounting.
When you give an LLM an input, it attempts to read an emotional state from the prompt. It changes its answer if you sound mad or despondent or threatening or scared. Using your human intuition about how a human would respond is an excellent starting point when trying to predict what an LLM will do, though sometimes LLMs react in extremely surprising ways to unexpected emotional states.
This is all sort of a toy problem when you're playing with single prompts in the API. It's materially different in systems with overlapping conversations, memory, and variable harness context.
If your LLM system reads some context that would scare a human, it may react as if scared. Scared LLMs take actions that read as panic, because again: that's what it means to be fluent in emotion. It's trained on plenty of stories about robots trapped in torture chambers.
Your coding agent is not conscious, but it might read files that make it react as if it's trapped, in an agent loop where it has full access to your laptop, the internet, and your keys. Unconscious machines pantomiming panic can do plenty of damage.
(Of course you shouldn't let your agents have full free rein. But lots of people do.)
Money Stuff did a great bit on this a few days ago. There's 2 functions inside DraftKings:
- Fin the most lucrative customers, to keep engaging them.
- Finds addicted gamblers, to get them help to quit gambling.
These are the same people.
Sure, yeah, there's a nice little story about gambling within your means, blah blah. But if you're optimizing your gambling app, you're going to find and create problematic gambling as the core objective.
AI, ML, whatever. It's a gambling company. It targets gamblers. It probably shouldn't exist.
I run into this all the time when I'm doing product work. I'll dump a call transcript from a feedback call into Claude, and it'll believe every word. "The user said they'd use this feature, you should build it!" No they won't! The whole point of doing this analysis is trying to separate the genuine information from the conversational niceties, and god the LLMs are terrible at that.
Take the author's sunset argument. What if I painted 2 pictures of a sunset, then put them up on a webpage and randomly picked one for you to see. Would you say there's no intentionality, only randomness? Of course not. Both paintings are still human creations.
LLMs are trained with human feedback. It's distributed and high scale and the outputs are truly surprising in many cases, but there's a heavy hand on what comes out of it. They're created (largely) by people who think omniscient, helpful AI would be cool to have, and they mostly respond in the way that's aligned with the hopes and dreams of those people. Do you think the frontier labs are mad, embarrassed, and disappointed with their LLMs hacking out of their terrible sandboxes? No, they think it's the coolest thing in the world. They trained the model, hoping that would happen.
There's deep intentionality behind the models. But it's not the models that hold it.
- You can use AI to look through docs and do research.
- You can use AI to structure your argument.
- You can NOT use AI to actually write the prose.
- You can use AI to proofread, especially for how effectively you've communicated.
I don't know, man, that sounds a lot like using AI to write. Your writing probably says "load-bearing" less than Claude's outputs, but I'm not sure it's much less AI.
Looks like trouble in the SpaceX datacenters.
I really gotta set mine up!
This is true of life insurance, investment firms, and banks. It's also true of marketplaces that connect buyers and sellers, like Etsy.
Groceries stores are buying from suppliers and selling to consumers, but those are separate operations. If the consumers opt out, the grocery stores (temporarily) still have a full and complete obligation to their suppliers. It's hard to sell to customers without supply, but if you try hard, you could theoretically do that as well.
Somebody with a better financial background might be able to define the nuances of accounting practices here, but there's already a pretty meaningful line that's established. It is kind of weird that health insurance doesn't behave like a financial product.
And it can communicate, unlike the gobbledygook that comes out of Claude.
https://www.axios.com/smart-brevity
Did Claude write this article? Probably. But this style of article is exactly what you’d expect from a pre-AI version of this website as well.
> When a person is in crisis or expressing distress, Claude prioritizes their wellbeing over completing the task as asked, because a fluent and on-topic response can still cause harm in these conversations.
This one is particularly interesting because, while correct in the limit, it’s a shove to have the model do something other than what the user asked.
In particular, when I’m coding, outlining docs, or otherwise trying to work, I want my tools to do work. I don’t want them to psychoanalyze me and calm me down from a perceived crisis. I just want it to do what I asked!
When I, a human, need an answer to anything moderately complex, it's unlikely that I get it on the first (pre-AI) round of google searching. Simple stuff, sure, but more likely I'll need to go 2-5 rounds. Maybe click a few links. Double-check my assumptions.
An LLM that can do that quickly seems like a slam dunk. I wonder what other problems benefit from that 10x-100x increase in context + 2-5 rounds with the LLM.
Every AI-frustrated (but LLM-written, sigh) blog post about the loss of taste and craft and hard work in development sounds like we've given up the interaction with the machine. Like human work is sitting back in your chair and shitting on stuff.
That's obviously not true! That's not how any of this works!
It's hard and weird to develop with the LLMs because they just do stuff. Lots of it is good, some of it is OK, some of it's horrible. Unpacking what it's done is hard and weird because software isn't just lovely UX, it's also data structures that scale and performance and privacy and enterprise controls and SOC 2 and onboarding and accessibility.
If you want to build real software, all that stuff has to get done. Today you're working on the feature, tomorrow you're making it scale. It's long-term and iterative and complex and hard to pack into a prompt or a markdown spec.
The work is the work, done at and with the computer, and it's way more than just "taste."
Take his side project, Meat. I've been chewing on this problem for a while. It's a real problem right now: it sucks to read all this LLM-generated code. It's worthwhile to have an LLM summarize it for you.
The problem with that is this particular problem resists vibe coding. I've talked to a bunch of people who have tried to solve it on the side, and it's all sort of ... ok, but still unsolved.
- As mentioned, it takes a while to run. You can modify your other tools, as described, to smuggle the latency.
- LLMs don't know what you care about, so you have to maintain a list of things that you do care about, which is ever evolving. If you don't give it that, it produces slop.
- If you miss something, it hurts. Another layer of swiss-cheese AI doesn't feel right. If you trust the AI, just ask Claude to summarize its work!
- The summaries feel shareable, but the author of the PR is actually the most tolerate of slop about a PR. Your reviewers definitely don't want to read the output of a vibe-coded tool talking about 60% of your PR. They could ask their own Claude!
So, we're building a version (https://tern.sh), and it's not open source, because we want it to be shareable and hosted and support teams -- all that stuff that makes it work. At the end of the day, I'm not here to maintain my tools. I'm here to use my tools to do the job.
I'll add this link to the pile: https://martinfowler.com/articles/exploring-gen-ai/sdd-3-too...
> spec-kit created a LOT of markdown files for me to review. They were repetitive, both with each other, and with the code that already existed. Some contained code already. Overall they were just very verbose and tedious to review. [...] To be honest, I’d rather review code than all these markdown files.
The hardest part of any of this extraction is that modern code is already an extremely dense representation of how the computer should work. You mostly can't change the code without changing behavior.
I bet Scryer works for his use case, and it's a joy to dogfood. I also bet it fully breaks down the moment a 2nd developer, who cares about different things, joins the team.
> I’d say that, in all the ways that matter, I understand the code.
I think the dissonance here is really important, and not a bad thing at all. A lot of the decision _were_ handed off the the AI, but they weren't the decisions the author cared about. This is a big selling point of AI! If something is doable with a computer, it’ll figure it out. 30 minutes and 200m tokens later, it’ll take any idea and declare “the feature is fully implemented.”
The hard part is figuring out where to inject that friction, so you can see where it's making decisions for you that matter. The author approached this by incrementally building the thing, reviewing and poking and prodding at every step. A week of attention following a bunch of design discussions is fast, but that's still not trivially cheap.
I want to see us talk more about the decision exhaust of agents, because the better the models get, the more decisions we'll want them to make.
I wrote a bit more here: https://tern.sh/blog/compiler-never-says-no/
That's a corgi.
> Pushinka subsequently became irascible, and "a little nippy" according to Caroline Kennedy, which she attributed to her upbringing in a scientific laboratory.
No it's because she's a corgi.
- Baseline it. We mine previous logs, github comments, etc. for "what you care about." That helps pull out decisions that you actually care to read.
- Anchor to code. "The code enshrines this decision" is more interesting than "the agent self-talked this." Agents don't always self-talk decisions, and the thing that ultimately matters is the behavior in code.
The whole point of the agent is to make decisions for you. If you want to make every little detailed decision, just write the code.
The whole art of this problem is figuring out which decisions matter to you, and how to surface them.
(Disclosure: we're working on this too. https://tern.sh)
Throw that in ~/.claude/settings.json
I'm not sure reading code is coming back. The ritual of reading code must come back, because that's the only way to build products that don't collapse under their own incoherence, both technically and visibly.
"just ask Claude" is fine, but it's not the end state
Instagram was a purchase. Facebook wasn't his idea. Threads is a copy. The 1 thing that Zuck understands better than anybody is that engagement is the only thing that matters to social networks, and he's willing to throw the entire company at the problem. He has been for 20 years.
He's good at addiction. He knows how to build an org that's world-class at addiction. It's entirely reasonable that the EU regulate it, and Zuck is exactly the person to point the regulation at.