azakai

> LLM code is abysmal if you look at the details

This is, unfortunately, false.

Yes, if you one-shot some code with an LLM, it can be terrible. But if you let it profile and optimize it, there is no obvious limit. LLMs have more patience to investigate performance issues and fix them than humans.

Again, I wish this wasn't so, but I see it in my job daily. There is code in my projects that is vastly faster because of LLM optimizations. Not only can they find more in less time, but they find things I and my co-workers would not have thought of.

For example, an LLM can prove a new theorem in math, verify it in Lean (so it is definitely true), and then prove more things based on that.

AI-generated data might not always be that useful, but at least in this case it obviously is.

It's even worse than that. Of course there is a mechanism in LLMs that could explain intelligence. That is the entire point of neural networks, from the 1950's! They were designed from the start as a model of brain computation.

Neural networks are not literally brains - just computational models - but if you are not a dualist, then computation is what the human brain does. Modeling that computation can explain something about intelligence.

Specifically: when scientists look inside a human brain, it seems it does its work using large numbers of highly-interconnected but simple units. The neural network model of brain computation begins there and tries to produce intelligent behavior. If it succeeds, then perhaps the model is right.

And it has succeeded: after 75 years, neural networks produce complex behavior that is arguably intelligent. Nobel Prizes were awarded. This does not prove the neural network model of intelligence is accurate, but it is a significant point in its favor, at least.

The author seems entirely unaware of any of this.

Also, you have to have a lot of confidence in the reliability of these systems to say, "If only OpenAI prompted 'do not hack outside systems' then the agents would not have hacked outside systems".

It would be great if they were so reliable, but I don't think they are!

Judging by the slow speed of node here, I wonder if it was run without forcing optimizations. It doesn't do OSR (on-stack replacement), meaning that a benchmark script doesn't get a chance to tier up from the baseline compiler to the fully optimized tier.

Running with

node --no-liftoff

avoids this problem.

(This is not an issue on the Web, where code must return to the event loop anyhow, allowing tiering up to work, and node is optimized for that kind of workflow.)

ChessNetwork on YouTube also does a great job with this: analysis of interesting games, using Stockfish as a tool along the way.

TFA says

> Type confusion in V8 in Google Chrome prior to 152.0.7977.82 allowed a remote attacker to execute arbitrary code inside the sandbox via a crafted HTML page.

So it was fixed in 152.0.7977.82 (before .83), if I read that right.

That their marketing department must love this does not prove it was intentional.

Those are fair points.

We don't know all the details about how the brain computes, you are right.

But we do have a hypothesis: that it is done by a large number of simple units with very high connectivity and in deep layers. This is what neural networks model.

Personally I was skeptical of this model of the brain, but they have achieved remarkable success in practice, as well as Nobel prizes. The neural networks people may have been onto something all along (I say that grudgingly).

But neurotransmitters and action potentials don't help in modeling symbolic structure.

That is, yes, ANNs are not brains. There are countless differences. But are there differences at the computational level? ANNs are meant to model brain computation, not brain biology.

(There is still a lot to debate there, I'm not saying "ANNs are perfect computational models for the brain")

> LLMs can detect patterns at a scale that no human ever could, but patterns only get you so far.

This is asserted without evidence, and from a scientific standpoint, unjustified.

First, "detect patterns" makes it sound like a classification task, "is this a picture of a cat". But LLMs transform the input.

For example, an LLM can translate text between two languages while properly handling the names of the people described, no matter what those names are. That shows they are representing the text in a somewhat abstract way, that they can perform operations on that representation, and also convert it to useful output.

And, what I just described is the most general form of information processing algorithm. Science is not aware of any limitations in principle on such systems.

I am not saying LLMs have no limits, but "they only recognize patterns, and that is a true limit" is not a good argument.

If you want a more concrete example, then LLMs are also trained on visual data these days, which means they do have access to the world in an important way. This directly contradicts the blogpost's claim that LLMs have

> no way to associate the text vectors they manipulate with real-world phenomena.

Historically, that LLMs were text-only used to be a major argument for why they "lack access to meaning", see the Stochastic Parrot paper and the Octopus paper that it references. But even the authors of those papers have (grudgingly) conceded that the argument no longer holds due to multimodality.

You can also take the opposite conclusion from this: C++ is now viable because LLMs can catch the security bugs.

I'm obviously not entirely serious here, but I think this is true to some extent: the number of memory safety bugs in a codebase is finite. Once you have a way to find them, you can drive that number down to zero.

C++ has become a far safer language thanks to LLMs - at least if you run the LLMs before you are attacked.

> How much of the identity of a mathematician is tied up in being the first to discovery?

Here is how I'd put it: math has an enormous focus on discovery. It is why we have Godel's incompleteness theorems, the Cantor set, Zorn's lemma, and so forth. We name things after their discoverers.

It is possible that, going forward, no more things will be named after human discoverers. The last such naming (of something truly significant) may already have occurred.

That is a massive culture change, at the very least.

Wait, what does this mean?

> We reuse WebAssembly linear memories under the covers to implement and sandbox the GC heap. A reference to a GC object is not a native pointer, it is a 32-bit index into the GC heap’s underlying linear memory [..] As far as being fast goes, it lets us use virtual-memory guard pages to elide explicit bounds checks, just like we do for linear memories

Array loads and stores still need an explicit bounds check, don't they? And struct loads and stores don't have one anyhow. Are there other bounds checks that Wasmtime is removing? I can't figure out what they mean here.

The overall point feels strained to me.

Yes, formal mathematics has such limits. We can't expect machines to be perfect and provably perfect. But the industry isn't assuming that. Why would it? Natural intelligence is not perfect or provably perfect, either.

Rather than certainty, measurement is often enough. We can't prove a program will always halt, but we can check it halts in a specific execution.

Approximation is also often all we need. Even if we can't prove that we can train a network with more than 50% success, if we can get multiple shots at that (using different data, or initial random weights, or training techniques, or something else), then we can reduce that danger exponentially. (I don't know that we have a guarantee of succeeding there, but this would be the hope, and I am not aware of anything showing it is impossible, unlike perfect provability.)

Finally, it is possible that perfect provability does work on the problems we care about. Godel and Turing etc.'s proofs rely on finding rare situations where we can't prove things - cleverly-constructed pathological cases - but perhaps human behavior does not fall into that set. Human behavior may not be a pathological case for proofs or learnability.

> All of the AI projects we have observed as a team are failing. Every single one – we have seen 0% success in a year and a half,

What is an "AI project"? The post doesn't define it.

Is it writing some software from scratch? Using an LLM chatbot by non-coders, either internally or externally? Or something else entirely?

Some examples would really help.

I'm not sure what "branch on undefined values" means there, yeah, but never reusing memory addresses is enough to prevent use-after-free.

Or, rather, you can use a value after freeing it, but it will not be exploitable, because it will contain valid data of the right type. This is the same idea as Type-After-Type,

https://dl.acm.org/doi/10.1145/3274694.3274705

(Also similar to when you use indexes to an array in Rust and happen to read from a wrong but in-bounds index.)

Zig does offer some amount of temporal memory safety.

Link: https://zig.guide/standard-library/allocators/

Text:

> The Zig standard library also has a general-purpose debug allocator. This is a safe allocator that can prevent double-free, use-after-free and can detect leaks.

For more detail, see:

https://github.com/ziglang/zig/issues/3180#issuecomment-5284...

Prior art: WebKit.js, the WebKit rendering engine ported to JS

https://github.com/trevorlinton/webkit.js/

The researchers in the field disagree with you. Look at conferences like NeurIPS and ICLR to see a steady stream of incremental progress in this area.

The optimism is based on the successes so far, some of which are described in this article. Scientists have made progress here.

Yes, we do see signs of actual reasoning, see the papers linked in the article. (There are many others too.)

Yes, we have a tendency to anthropomorphize, but (most) researchers are aware of this.

The article answers this question, at least to the extent it can be answered, at this time.

We see some signs of reasoning, but also we understand little about how they work.

They made a claim about language models in general, not just ones that had been released so far.

The point of the paper, in fact, is that language models are getting "too big", and another approach is needed to make progress, so they were certainly predicting things about later models.

With that said, they talked about "pure" language models, so it is fair to say that they didn't talk about, say, LLMs that are multimodal or that have tool use, which are advances that happened after their paper.

My main criticism of the paper is that it says LLMs work "haphazardly", using probabilistic information. That is a hypothesis, but it is stated as a known fact, a fundamental limitation.

It is true that LLMs often behave haphazardly, and do rely on statistics. But plenty of research has shown them behaving in methodical ways too. There are findings going both ways!

Granted, many of the strongest contradictory results appeared after the Stochastic Parrots paper, so it isn't like they were ignoring the literature at the time. But they did make a very strong claim, and in the half-decade since, a lot of evidence has come out against it.

Yes. Godot and Zig are the exceptions, and therefore newsworthy.

> Let's keep WebAssembly lean and fast!

Note that wasm is still lean and fast - WASI is not part of core wasm, but layered on top.

That is, it is possible to implement wasm without WASI. That is also true for other wasm proposals like WasmGC. It is very possible that parts of the ecosystem will not implement certain proposals if they don't make sense there (e.g. parts of the embedded ecosystem may never add GC, etc.).