WhitneyLand
When good expositors choose motivating questions, say like 3 blue 1 brown, they don’t usually make apples to oranges comparisons. The best examples motivate while making sense before and after you learn.
Mostly the trick is they're comparing lossy to lossless compression which doesn't even make any sense.
Find it useful to conjure up cognitive dissonance and practice letting it go.
- Text interface: Done
- Voice interface: 5-10 years I’ll predict
The demo is nice, but enough obstacles remain that very few people would mistake it for a human over even a 5 min conversation. Chat systems configured to intentionally impersonate can already get people to mistake them for humans most of the time.
There’s been great progress in this area, direct response modeling, parallel reasoning, but last challenges are huge.
The reality is LLMs still seem dumb much more often without significant reasoning time (test time inference). On that point alone they are still an order of magnitude too slow for convincing interactive latency.
Of course they don’t need to meet this bar to be useful, systems are good enough today for lots of applications. It will just be interesting to see what the final delta ends up being.
However if my new car's engine is a near copy of the one in the Corvette, and my announcement gushes about the engine using the same terms the Corvette design team invented...
Then can you see how a mention might be nice?
It’s not meant to be an engineering improvement, it’s an important theoretical result because it casts doubt on things that were previously thought to be impossible.
For my Figma needs, having Codex do computer use seems just as good as their mcp. I can tell it, “go download the assets for what I need and take a few screenshots for reference”.
- Maintaining a significant new moving part in the dev process. KMP updates, management, integration, futures awareness, this all takes non-zero time.
- If an iOS dev wants to work on a KMP project what's the ramp up time? How much less efficiently can they contribute compared to a pure native app?
- How much time is spent on problems, bugs, issues, related to KMP. This is guaranteed non-zero for any tool.
- Is there less benefit to KMP with coding agents? Previously with two native apps, a bug fix had to be done twice. Currently, say a fix in done on Android, it's quite easy to say "propose an equivalent fix for the iOS repo". I do not claim this removes all benefits of a single code base, just that some things are not as bad now.
So I’m sure they won’t be fixing it then.
They are often too verbose, or miss capturing an important concept or purpose, or add bullets for parts of the changes that no one cares about.
This comparison doesn’t even make sense.
Lots of papers have great results that don’t depend on the latest models.
However in this case it’s problematic:
- They specifically make claims about the state of “current LLMs”. o3 is not representative of this.
- They ask are LLMs capable of X and arrive at a negative result.
If their claim was LLM’s can write coherent sentences, and their conclusion was positive, then there would be no issue using old models because the end result would be factual.
However, when you have a negative result that makes a claim about the current state of all LLMs and the ones you were using are not current, by definition it draws the whole conclusion into question.
2. Chess seems to be a poor benchmark for generalized strategic reasoning. People who are good at it rely more on experience and deep domain expertise than on skills that generalize to make them experts at unrelated tasks.
3. The study sounds like proving humans will never fly because they don’t have wings. In reality, humans do fly, and Claude Fable would destroy any human at chess by coding a strong enough engine on the fly.
- Give Jev and LLM the same input
- Lock down both to approved/rejected/unknown (LLM restricts on decoding)
- Both can be wrong, but neither can hallucinate (invent an another option).
"Jev: New frontier model 40-400x cheaper and 20-200x faster"
I'm not the gatekeeper of who gets to call themselves a frontier model, but I don't think most people would count Jev in that group. It sounds false.
If their specific claims hold up, then it would make more sense to say something like:
"Advanced the speed/cost frontier for structured decisions"
The original title before it changed less than an hour ago was:
"Jev: New frontier model 40-400x cheaper and 20-200x faster"
I'm going to agree that was misleading.
And on the second point:
>>Also "can't hallucinate" seems wrong? Sure, it can't emit an invalid type, but it can still emit a completely wrong valid value.
>that is likely true of all ML! perhaps we could debate semantics, but I don't think it's fair to say a random forest "hallucinates" in the way LLMs do"
Also going to disagree here, and I don't think it's semantics.
Type safety is not factual correctness.
Any attributes or keywords relating to Objective-C or interop are not everyday baggage for most people.
For SwiftUI, attributes seem like as good a choice as anything else.
As a whole concurrency is a mishmash syntactic mess, but I agree with the direction and the result, they're trying to manage the improvements they keep adding over years of investment.
To buy presents for a family Christmas list Mom drives to Store A and Dad drives to Store B.
As more items get added to the list, they must decide who should drive to a new store location to buy the present. How can they minimize total driving distance while kids are randomly adding new items to their list?
The proof above guarantees its possible to never drive more than twice the mileage you would knowing all the items in advance.
The big news is this guarantee works for any number of drivers with any arrangement of gifts.
The algorithm to do this was already known, what we’ve learned is it’s not possible to do any better.
One way to think about competitive analysis is bulk discounts. In life we’re constantly having to choose between quantity and discount. We could buy 1 item for a higher price, or say quantity 5 or 10 to get better discounts. The problem comes when we don’t know in advance exactly how many we’re going to need.
What should be our strategy for choosing how many to buy, and whatever the strategy is how well does it compare with having perfect knowledge upfront?
I mean, I otherwise agree with the sentiment of your post but I feel like most people are rating him pretty well.
In your generalized example I think the concern is when the additional evaluation effectively becomes a replacement for CoT, where something like the coconut research could replace it completely.
However, I don’t think we’re anywhere close to that with Astra.
Looping transformers uses additional calculations (repeating layers) to generate a token.
Reasoning (in this context) is test time generation of multiple tokens that allow a model to have a scratch pad to refine its thoughts, chain of thought reasoning in other words.
Doing the former in no way means that you have to hide the latter.
Raschka is right in this post, The Information article was wrong. The Astra system card does concede reasoning traces are sometimes smaller, but this could be for a lot of reasons, including simple efficiency. And it absolutely doesn’t mean they are going away or completely obscured.
The Last Week in AI podcast from Sept 8 seems to have gotten this wrong as well. Jeremie Harris rages that OpenAI implemented latent reasoning, ala the coconut paper, which could potentially actually obscure reasoning traces. But for the life of me, I do not know how he arrived at this conclusion and see no evidence that this has happened in Astra.