jackb4040

I don't know how much we learn by surveying working "AI researchers". By their own admission most of them have a religious belief that they will bring about a singularity in the next few years. It's like asking a billionaire if he earned his wealth; regardless of what you think the answer is, you pretty much know how they will answer ahead of time.

What it shows are two unrelated results mixed together for no apparent reason besides marketing. The raw-weight results are impressive, but haven't been demonstrated as ready to scale up to a real language model. The Qwen result is a mildly interesting application of functionality that's been on the market for years.

If you think that the definition of "RSI" includes systems with a coding agent in the critical path, something humans can already do albeit faster, then your definition includes all of human history. If your definition includes AI systems that learn in a similar way to biological intelligence, then only the raw-weight results from this paper are pertinent. Those results are indeed interesting! This article is merely disentangling them from the frontier-lab-fueled hype that overran yesterday's thread.

It obviously is an important distinction to the paper's authors, which is why they used the modifier "native" to mark it as a distinct form of communication than what's already been on the market for years (coding agents). But it's not native, or rather some of the results were native but they mixed toy-scale native results with non-toy-scale non-native results for marketing purposes.

This is like someone telling you they made a "native" brain-to-brain interface but it's just a tin-can phone.

Gotta get to the IPO, whatever it takes.

It saved the stock market. Try telling one of the thousands who was made homeless that it saved the economy.

The USSR is the superpower I was referring to. You are illiterate.

The USS Liberty happened in the 1960s. It's downright silly to pretend the American state would ever have had a similar orientation towards a rival superpower vs its own unsinkable aircraft carrier.

Well it's important to you. It's evidently not important to the people you vote for, the people who fundraise off of this hostage situation, the people who have had trifectas multiple times during this era and have not even used them to permanently secure the "democracy", let alone the overwhelmingly popular policies with their base that would guarantee they win ever election but piss off their donors.

> half

They own the careers of all but a tiny handful of US politicians

It's hard for me to understand Google's redefinition of "world models" (basically: a txt2videogame modality for LLMs) as anything other than an attempt to bury Yann Lecun's definition of "world models", which includes an implicit bear case against LLMs and thus Google's stock price.

From the perspective of your career, the best thing to do is not have objections, or really any principles at all, in the first place.

This is so vague as to be impossible to pin down what you mean in order to refute it without you slipping out of it. But in very general terms, trust busting was a real thing that happened in this country once upon a time, yet would be unimaginable under the current ruling class.

Why would you expect training costs to come down in the future? It seems to me that the labs themselves have concluded they can't stop training due to competition, which is why they're now asking the government to step in and shut down their competitors.

How do we know this isn't just propaganda to force people to learn about the natural world, go outside, and make new friends?

It's a shame you've been downvoted, this is better satire than anything A24 has put out in years. Must be a large number of people who saw that post as a slightly exaggerated mirror held up to themselves and didn't like it.

I have a pretty negative opinion about LLM's coding abilities (and predictions about how they will improve). But I don't get a lot out of this type of argument. Compilers created a perfect, deterministic, provable abstraction so that we could have 99% of developers no longer know how assembly works and not suffer any consequences. Humans don't have to give up any control or make any quality compromises to deploy them as widely as possible.

If the author is contending that AI models will improve to a point that they actually make no mistakes while coding, and have that same level of reliability, then they're on exactly the same side as AI doom trolls just wearing a different costume.

Agree, and I am worried about what happens when that old code is written by LLMs, approved by a human who pretended to have read it, and now needs to be touched by another human who is completely reliant on LLMs to understand existing code.

Seems like a non-sequitur? "because math is formalizable", I agree it makes sense to then predict that data-driven AI models will be able to perform well on it. But where is the evidence that software coherence is formalizable? If it were then I would agree, but to me it seems that coherence in a software system relies on a ton of intuitive reasoning and familiarity with company culture, past failures, etc. it's like the least formalizable thing there is.

AI improves over time, sure, but it has differences and advantages and disadvantages over human intelligence. It's just a fundamentally different thing. In the same way calculators are superhuman, and improving, but can't write books. You yourself acknowledge to some degree the uneven nature of that improvement when you referenced formalizability.

Fantastic framework that I will be adopting, thank you!

One thing I will add: I'm sure the optimistic side will try to take the quantization that your (and other work like SlopCodeBench) are introducing and claim that we'll be able to fix the problem by training on that data, producing coding models that are rewarded for coherence.

I invite them to try! But my suspicion is that this will require a rather deep form of intelligence (world models) that the current language/data-based paradigm isn't really equipped to reach without requiring more energy than the Sun.

In the day since I posted this, I've seen a few good links on HN that touch on this question.

1. The Decoherence Trap: https://maevyn.codes/decoherence 2. SlopCodeBench, a benchmark that quantifies how models decohere over repeated change requests rather than a single one-shot challenge: https://news.ycombinator.com/item?id=50108717

I think the most important takeaway is that it takes time for these problems to manifest. If you start with a particularly coherent codebase it will take years for slop code to degrade it to a point where engineers are noticeably less productive.

I didn't downvote you, but if I had to guess why you were downvoted it's because it's very obvious to many working software engineers that decoherence is a real problem. In many ways that's what this OP was all about; not just loss of joy in programming for its own sake but loss of the ability to design deterministic, fully-understood systems that won't require constant off-the-books maintenance eating into your free time.

No, they're in post-colonial conditions and intentionally prevented from developing economically via a relationship of economic dependence on the imperial core. And as a result, they can't produce their own chemotherapy, insulin etc. and have to import it from developed parts of the world system.

If the imperial core spontaneously disappeared, they would begin developing - and that development would take the form of centralization and specialization and state-driven investment in underdeveloped sectors. Look at what Burkina Faso is doing right now for example: they went from contracts with French companies to export gold and uranium for a thousandth of their market value to now climbing the value chain by developing their own gold refineries.

To keep the example but circle back to the original point: If you lived in a future Burkina Faso that successfully developed its economy, proposing that they go back to small-scale production with worse economies of scale would be a step backwards, and yes the logical conclusion of that in any country is hunter-gatherers. There can be other important learnings from doing this as a constrained experiment within a development model, like resiliency. But thinking this is a development model in itself is a cargo cult mentality that misunderstands industrial production as some evil external imposition on humanity rather than the system that proved most effective at feeding the most people.

Those companies, and those people within otherwise decent companies, have existed since long before the advent of LLMs. Avoiding their influence is a career skill as important as the code itself. And I'm not saying you can just avoid them for your whole career, they're everywhere and are being supercharged by LLMs.

It's disappointing that the prevailing sentiment now is that architecture is less important than we thought. Software is always going to be getting more important, and all of it should be well thought out. If these people had these tools in 2004 they might've made thousands of fragmented buggy social media startups and actually slowed progress on that front.

Thank you! It's made me lose a lot of respect for my field how credulous everyone is to any argument that starts out with a fig leaf of pretending to be on your side.

"The AI labs are scared of their own models, surely they must be credible sources on how powerful those models are"

"They are asking the government for regulation, they must be even more responsible than the average company"

"They are claiming to be a software engineer who automated their job with AI, surely I have to buy their course"

There are many boilerplate pieces of software that an LLM can generate faster than a human, but nearly half a decade into the LLM era there is no evidence that it's leading to the apocalyptic end of software organizations. And that's all with the subsidy in place that is now disappearing. Everyone online is lying for money, and that money is coming out of the same investor-subsidized marketing budgets.

I think you're entitled to your opinion, but want to point out that there's 2 different axes on the graph here. You are unhappy about LLMs' affect on your working conditions, but also concede it's inevitable and the labs are at least somewhat sincere in their predictions of it displacing a significant amount of the value in the software engineering market.

It's not necessarily a given that the person you're replying to agrees on the latter, and therefore they might not be stoked on you starting off by agreeing with them but going on to make an argument that supports the companies responsible for the very thing they're complaining about.

I think there's a contradiction in you stating that management thinks LLM-generated output is "maintainable". They went to business school, they don't know what that means. It's your job to explain to them the consequences of shipping code faster than the people whose job it is to understand it can do so.

> The ability of people to freely associate

> solipsist

This is exactly the point they're making though. This is an exercise only available to the privileged. It is not available to people who haven't worked in tech for some amount of time, if not actively working in tech or holding equity like the homesteader above us.

It takes for granted the concentrated specialization of production that is modern, "on-grid" cities to produce things like microprocessors and food safety and insulin. This lifestyle is not available to masses of people, and we have a word for a society where everyone lives off-grid: "hunter-gatherers".

There can be other value in doing this, like Hundred Rabbits' exploration of how tech itself could be made more resilient given different macroeconomic conditions / incentives. But talking about this as if it's about how The Man just doesn't want you to be an independent free thinker is a misunderstanding of why most people live in the societies that they do.

LLMs are very useful, I use them every day as a software engineer to solve problems and search for information represented within the data available to them. But they are a specific type of intelligence, with many advantages and disadvantages vs human intelligence and it's not clear that just scaling or tweaking them without a theoretical, architectural change will make them more generally intelligent than humans (despite all US AI companies promising exactly that).

They are fundamentally based in language, and achieving deeper models of the world through language alone is deeply inefficient compared to the way humans model the world for years without any language at all. They do not learn at inference time. They don't have semantic understanding of the difference between their own output and other sources. etc etc.

That depth is the key for me. Of course they are capable of producing novel sentences that aren't in their training data, but the depth of that novelty is basically within the bounds of language itself. They are capable of more serious depth and more abstract reasoning than that, but I have experienced limits, which it then tries to surpass with tools to convert things it can't understand back into language (unit tests, LEAN) upon which it is trained.

Because I'm not an AI booster, my account is limited to 5 comments a day. So this is the last reply I'll be able to make today, if you want to continue the conversation we'll have to wait for tomorrow.

Open sourced by whom? Was this a standalone company or a spinoff of Figma or something? I imagine they must have been making serious money to afford that domain.

I do hobby work in Three.js and have always felt more comfortable using the declarative scene.add()-style API to manage my scene vs fiber / reactive stuff. But I imagine I'd feel differently if I were making 3d components tightly integrated with the state of the rest of a normal react app, rather than a fully immersive game scene.

I'm glad both apis exist.

But doesn't that just amount to labs intervening to teach the models to use a particular strategy to mask this one very obvious marker of the difference between their intelligence and biological intelligence? (And similar surface issues like using tool calls / reasoning for arithmetic, even though humans writing on the internet don't typically break show their work for multiplying two numbers)

The deeper architectural difference is still there, which manifests whenever you try to get the models to apply known techniques to modalities and problems outside their training data.

I don't quite understand the leap you're making between stochastic AI models for robotics (maybe with an LLM making api calls to it) and embodied AI / the rate of progress towards a singularity. Because they're both trained on a GPU? Up until 2022 GPUs were for video games and mining crypto, and neither of those produce a synergy that accelerates progress towards general intelligence either.