soVeryTired
soVeryTired on Hacker News
That sounds like a next token generator to me, which I think is more accurate. The word "predict" is doing a lot of heavy lifting in your formulation. What exactly is being predicted?
Yeah, fine. I was using the word design loosely. I'd also happily say something like "humans are designed to like salt and fat" and feel like that was a pretty reasonable statement, even though humans aren't designed either.
At this point it's either wrong or meaningless to call LLMs "next token predictors". Yes, pretraining involves predicting the next token. But after that you're in the world of RLHF, RLVR, and other post-training.
A model is _sort of_ designed to predict the next token in a string of text. But it's equally accurate to say it's designed to edit code until it passes a suite of unit tests.
Yes, they generate sequentially, so "next token predictor" isn't technically incorrect (which is why I chose the word "meaningless"). But it's pretty clear there's a mechanism inside them that allows them to "plan" what they generate. I might as well call you a next word predictor.
Does Iain M Banks count as pop culture?
Far more than 500 person-years have already gone into Navier-Stokes.
Or maybe the PhD impersonator discovered a soundness bug in a component of Lean and was able to exploit it.
With no humans able to understand the proofs the impersonator has produced, there's no way of knowing.
Not sure I agree - they dropped hundreds of papers, most of which were so badly written that they were incomprehensible. If it wasn’t for the lean certificates on 30% of them, the mathematical community would have told them to go and whistle.
I think the consensus here is that most of the papers are effectively correct. But it would take years of study just to understand most of the problem statement - so we have to wait for the mathematical community to digest the borderline-nonsense that OpenAI spewed out.
But up until now, the mathematics community has valued the "prove" part much more highly than the "deliver an insight" part. Mostly because with a little work they went hand in hand.
And going from zero proofs to one proof (even a sloppy one) is a big deal regardless of whether it was written by AI or a human.
Probably true for the dedicated problem solvers (of which Tao is one IMO). But I doubt there's ever been a better time to be a theory builder (more like Peter Scholze, or Grothendieck).
Some up with an idea and leave the system to check it 15 different ways, and see whether you can simplify an existing body of theory. It'd be like having an army of lightning-fast grad students.
What's an example of your belief system conflicting with a motivated example? I'd love to understand that a bit more.
Wintermute and neuromancer!
Was it a marketing win though? My takeaway is: if you're doing groundbreaking work with openAI's models and they find out, at best they'll outspend you and scoop you. At worst they'll steal your chat history.
Yeah - I think he’s worried that we have a tool that’s
almost an oracle that can just get straight to the heart of a given problem.
The problems aren’t always that interesting in their own right. But the quest to solve them is often what drives the invention of new techniques.
We ultimately got the Langlands program and a large amount of algebraic number theory out of attempts to solve fermat’s last theorem. A “clever” approach using techniques from the 1800s wouldn’t have been anywhere near as fruitful for mathematics as a discipline.
Out of interest, would you call BERT an LLM? It’s pre trained but not particularly large.
It’s neither large nor language-based. ARC-AGI-1 is grid-based and nonverbal.
Use of a transformer is not necessary or sufficient to qualify as an LLM.
Undetectable by a human reader. Come on, give the post a charitable reading.
Yeah - otherwise it’s bizarre to have row vectors and column vectors.
To that point, given a corpus of writing from person A and another from person B, I have no doubt it’s easy to train a classifier to determine who wrote what. In fact I believe law enforcement agencies already have these classifiers.
The only difference here is that Anthropic is actively trying to make the watermark undetectable.
Out of interest, do Musk's politics impact your decision on whether or not to use Grok? I'd be interested to know where folks lie on the (Agree / Disagree) and (Use / Don't use) axes.
0000 in base pi. Oh Jeff.
I mean yes but where does that get us? The great oxygenation event wasn't exactly good for the organisms that were alive at the time.
No way is vocab size zipfian. Word counts from a corpus follow zipf's law, but not vocab sizes themselves.
Otherwise the most common vocab size would be equal to one.
Can anyone explain to me why Q and K are both needed? They only ever appear as a pair, so why can’t you just define a matrix A = QK and learn that directly?
> On March 12, 2025, a search warrant was executed at Sohaib’s home in Alexandria. Agents grabbed plenty of tech gear but also turned up seven firearms and 370 rounds of .30 caliber ammunition. Given his former crimes, Sohaib should have had none of this.
For god's sake, don't commit crimes while you're committing crimes.
I have a feeling taxes are possible to circumvent only because a government tends to have one arm that wants to collect taxes, and another that wants to reduce them to encourage certain outcomes (like having a business setting up shop within its borders).
The US may have this dual incentive structure since it wants to build its tech giants while limiting their control, but the EU doesn't. The arrival of a foreign tech social media giant might make the legislation a bit more palatable to pass.
It will undoubtedly be complex to regulate all dark patterns away. But there are a few obvious, easy wins. It'd be a shame to make perfect the enemy of good.
Us consumers don't want moats. But to understand the investment you need to understand the moat: no sensible investor will throw this much money around without one. If you don't understand what is keeping new players from entering the market, then either you've missed something or we're in bubble territory.
That's why it's interesting to try to pick apart where the moat is.
Whenever people bring this up I like to remind them that linear interpolation is a universal function approximator.
The company is named after the evil telepathic orbs from lord of the rings. Wasn't that the first clue that everything might not be hunky dory?