jonahx

> That is the public way he navigates the situation, who knows what his real game is.

What his "game" is? I've been following Tao for years... I can't think of someone I'd be less likely to accuse of having some nefarious "game".

> He is one of the people that brought about this situation.

A baseless accusation. Tao has no power over the progress happening at the frontier labs.

Yes, you're right, understanding informs the constraints and the possible solution spaces. That should be added to the list.

Practically speaking, I don't think the "black box" metaphor is as broken as you're saying, though I like the point you made. There is still a ton of software (the majority, I think -- crud apps of all flavors) where an unthinking, untechnical PM wants something they can specify, and an LLM can give it to them, and it will more or less solve their problem. It won't be the best possible thing, but it will be good enough.

I don't disagree with the points you've made but what are they rebutting?

The economic value is still the product you delivered. Your understanding of the implementation meant you participated in the work of the "product manager". I think most good devs do. That might be an argument for devs to transition into designer/PM roles. That is effectively happening already, whether or not role titles have caught up.

But in any case the value isn't in the understanding itself. It just happens that the understanding still helps create better block box products. Also, I wouldn't count on this "human advantage" lasting long. I hope it does, but ime LLMs are now good enough to do this "co-design" too, and will only get better.

The parallel is real, but in math the "cash value" of the results often won't appear for years, if at all, and so the teaching and "intellectual fortification" is the primary point of the whole enterprise (at least, a major part of it).

In contrast, the value of a program is typically just the "black box" service it provides. Indeed, the program itself has always been opaque to most business stakeholders. Before, they paid human programmers and got back a program they didn't understand but could use; now, they pay an LLM provider and get back a program they don't understand but can use. In the pre-LLM era, the economic value of "understanding" and "shared knowledge" about the programs themselves was only that it made maintenance and changes cheaper. It never had intrinsic value, except as intellectual satisfaction to programmers (I was... am... one of those who found that value).

> Limited in what way? Its basic structure is sufficient to eventually let us discover anything there is to discover. It's a fully general problem solving framework.

That "eventually" is doing a lot of heavy lifting. It's limited in the most literal, empirical way. We don't have answers to most things. If we stood in relation to a gargantuan code base as we stand in relation to the natural world, with an equal proportion of known vs unknown, we'd be pretty screwed.

As an aside, even with the "eventually" qualifier, the claim isn't true. The generation of hypotheses, in particular, and even sometimes the experimental setups to test them, happens through an "out of band" process -- namely, human creativity and intuition. Which is effectively a computational process we don't understand, and limited in its own way (despite its incredible productions). The history of scientific breakthroughs is the history of astonishing insights out of nowhere, from the brains of individual people, sometimes literally from dreams. The experimental method provides a reliable way to test the insights more or less "mechanically", but it would be mostly impotent without the other part.

> we have effective tools and processes for each of these search spaces

Do we? Science (the tool for the natural world) is limited and there's so much we don't understand.

The recent superhuman achievements by llms in math are promising, but what we've seen so far isn't enough imo to YOLO building arbitrarily large systems in code and just trust that the llms will be able to handle it. Especially when the stakes are high (human lives, etc).

GP is saying that "the search space" of "the natural world" is much bigger than the search space of "stuff created on purpose by humans". In both cases, there will be guesswork, but in one case less.

I think there is something to this, but am even more concerned by the pure scale. Consider 100k, 1M, and 10M LOC human legacy projects (already a big problem, usually!), and now compare to the likely outcome of 100M, 1B, 10B, etc LOC llm projects. That's where you may start to see the outcomes I reference in my OP.

Right, that's what I alluded to in my last sentence.

It is true that this gamble has existed in the form you mention for many years. Still, I sense something new here, both in scale and in the attitude of trust. Also, I find those massive, incomprehensible human-built systems frightening too.

> have the LLM focus and summarize a slice of the system when they need to know about it or ask for it. But.... That's not knowledge? That's just search a la google? And thus seems to be subject to the same problems where the quality of your search dictates the quality of your knowledge that you surface at any given time?

This is the crux. Take pause, because hidden here is a sea-change in our fundamental trust model of software...

There is no longer anyone to turn to if this system breaks by, eg, having a critical bug that no one can figure out. No causal model in anyone's head that can be appealed to, to slowly and methodically debug. By design, the system is too large for that, and everything hinges on Claude itself being able to find and understand the right slice of the system in any situation. As long as that very new invariant holds, everything will be all right. If it fails catastrophically, there is no recovery. You are the owner of a recovered alien spacecraft with tech you have no hope of understanding.

The new trust model is: We trust Claude to always understand the system enough to debug anything or make any reasonable change we could want. The risk might be worth the reward. But it's worth explicitly laying out the deal we're making.

Tbf, it's worth pointing out that many large legacy systems already work like that (sort of), with "Claude" replaced by "whoever happens to be maintaining it this year".

We seriously looked into Alpha School, my impression was strongly "overpriced con". Frankly, I don't believe their marketing material, which is what you're quoting. I am not anti-AI in learning (LLMs have a great learning boost for me personally), but learning especially for children is a complex problem and truly making it more efficient will take more innovation than an ipad with a magic program. I would caution more skepticism with these results, especially self-reported ones.

You can honestly acknowledge what both industries do without "being out there trying to ban them." In any case, plenty of people want to improve working conditions in the agricultural industry. Probably the best you can do in both cases is imperfect regulation which at least tries to change incentives for the worst kind of abuse.

I never argued to ban pornography, and I'm not in favor of that.

That said, no, it's not like other industries. The amount of well-documented abuse is much higher. Degree matters.

This is not the claim. The claim is that, empirically, many (or enough) of the women who end up in porn are underage, trafficked, or psychologically coerced (due to poverty, drug addiction, abuse, etc) such that the choice is not really the free choice of willing participant.

I am all for individual freedoms, but I've read enough about this issue, and seen enough documentaries, that I think the claim is very well-founded. Of course, some porn stars are just willing adults doing something they want to do, but there are many who are not.

I don't understand the relevance of time to fix. It has no correlation with severity.

The headline here is that none of the bugs were serious.

I've never heard it referred to as "The Outsider" before. TIL it has been published under that name, but "The Stranger" is the title it's typically known as in English:

https://en.wikipedia.org/wiki/The_Stranger_(Camus_novel)

> yeah, the llm approach is incredibly wasteful wrt pretty much everything.

Everything except what matters most: human time.

GP is correct. Very few people were capable of even the informal analysis you are describing, and fewer did it. I'm not saying it's not valuable... just stating that, empirically, it rarely happened.

> They changed course and put the land into collectives and it is still in collectives today so the famine cannot be blamed on collectives.

Painting it as if Mao era collectives are the same as the state quotas that exist today is highly misleading. The HRS was 1979:

"The household responsibility system replaced collective farming." [1]

Broadly speaking, the collectives can and should be blamed for the famine, just as in the Ukraine. Getting into the weeds on the specific mechanisms is fine, but using it to undermine the broader lesson is apologetics.

[1] https://en.wikipedia.org/wiki/Household_responsibility_syste...

> I'll let you know if you're accurate or not

I think not. Your framing thus far is so wildly inaccurate that I believe this is a propaganda account, and hesitate to engage further. But in case any not familiar with the history read this in passing, I will leave a reference to an actually reputable source:

"Mao’s violent collectivization and forced labour campaigns during China’s Great Leap Forward (1958–1962) led to as many as 45 million deaths in what is widely regarded as the worst famine in human history." [1]

[1] https://www.cambridge.org/core/journals/china-quarterly/arti...

> shake up the distribution of it so that the land could produce again... It's how you turn a billion serfs into a billion entrepreneurs.

In your telling, how does the resulting famine that killed of tens of millions fit in?

crop-maxxing

We need a term for the dark pattern of zooming into just that part of the y-axis where the two closely competing benchmarks sit, to make the top one appear maximally better.

I don't think this has any teeth. They don't compensate in the event of loss afaict.

Thanks for clarifying.

> Ultimately, we have made it

Spinning the present situation or the history of nuclear arms as a high-five, "go team human!" success story is... quite the take (Vasili Arkhipov, Cuban missile crisis generally, the current doomsday clock being "the closest the Clock has ever been to midnight in its history").

> You don't have help. But perhaps you could at least avoid discouraging people unnecessarily?

More germanely, you don't have to worry yourself, but at least avoid discouraging people with legitimate worries who want to take precautions. Even the present situation with nuclear weapons, precarious as it is, would likely be more precarious were it not for the political pressure of the people worrying in the 1950s and 1960s and up to today.

It's funny that seems like a step the human mathematicians would want, and (at least for now) might still outperform the machines on. In the same way that, eg, the notebooks of Galois contained the core breakthroughs in a messy form, and generations after him simplified and synthesized those ideas, until you finally have books and videos accessible to undergraduates.

The concerns seem valid.

I'm unclear what the ask is, though. What, even in theory, is a practical and realistic fix?

Failure to imagine an incentive doesn't mean there isn't one. You can't rely on this type of thinking to reason about security. The thing you would have never thought of is what gets you.

For example, an ad provider itself can be hacked by a malicious party, so the "pay money for" part no longer applies.

Or an attack by a state actor or other large entity, where paying for a coordinated disruption of some region or company makes financial or military sense.

Those are just two things that came to my mind, and are likely a fraction of plausible incentives someone might have now or in the future. People are creative and unpredictable. Weird shit happens. Fact is stranger than fiction...

Instead, just ask: should visiting a website ever have the power to freeze your computer without your consent? If you think the answer is no, this is a security bug and it should be fixed.

I'd guess GP id enraged that you can't selectively choose, or that there is no option to have the tabs restored without auto-loading, so you can keep your list of open work without being forced to reload a freezing tab.