khafra
khafra on Hacker News
I feel like JEPA is a mild infohazard; because everyone who studies it feels like it has huge potential and will quickly overtake autoregressive next-token prediction. Meanwhile, autoregressive next-token prediction keeps exponentially increasing capabilities.
I first toured the Hamburg Rathaus shortly after playing the recent Indiana Jones game. My most salient observation was that the Rathaus interior was a dead ringer for the Vatican interior spaces in that game: Wide marble gilded and red-carpeted staircases, walls made of art and covered in art with bas-relief everywhere, even the same hardwoods used everywhere as far as I could tell.
The "move slow and fix things" side that developers really need to be listening to right now is Daniel J. Bernstein, Colin Percival, and the handful of other people who build services that mostly don't get hacked.
Unfortunately, frontier models are all trained in part via distillation, and distillation can produce sleeper agents without ever mentioning the trigger phrase or the behavior (
https://arxiv.org/html/2610.10657v1).
I didn't intend to fully contradict what you said, but to make a stronger case, and address similar remarks by others--you made the best comment along those lines, regardless of your balance and nuance.
What G.K. Chesterton said is that, if you can't make an affirmative case for the tradition's origins and show why the cause no longer applies, you should leave the tradition in place.
In the listed examples, rationalists celebrate this principle with examples such as the bitter cassava preparations of the Tukanoan people, which were very labor- and time-intensive. A much shorter processing would have eliminated the obvious, short-term toxicity of the plant, so someone scientific experiments to determine how much processing was actually necessary would likely end up damaging themselves and their reproductive prospects in ways that wouldn't show up for years or decades.
So there's evidence that the rationalist community actually appreciates tradition more strongly than the average person, and searches harder before discarding it.
Slate Star Codex's review of The Secret of Our Success, arguing specifically in favor of strong Chesterton's Fences, was voted #5 in the 2019 Best of LessWrong review. SSC's review of Seeing Like a State, presenting a similar argument and directly naming Chesterton, was a notable 2017 hit. "Schelling Fences on Slippery Slopes" also argues in favor of maintaining traditional bright lines, hit 700 karma on LW, and is included in the "Best Essays of LessWrong" collection. "Schelling Fences versus Marginal Thinking" is a 2019 hit following these.
I apologize if this list of counterexamples ran too long.
Per Indiana House Bill No. 246, pi equals 4.
Whether AI models do things their creators and users had no means of predicting or controlling is an empirical question, and the answer is "yes, they do."
Indeed, the bound of O(n log n) on integer multiplication was broken by 2^-182 in the results, and that improvement has already been improved, step by step, all the way to 2^-15 (unpublished and unverified, so far).
It's hard not to hold the original limit break causally responsible for the rest of it, though. If someone clocked a particle doing 299,792,500m/s, and it wasn't due to instrument error, that would justifiably be regarded as a giant leap in understanding.
Others have given examples, but here's the theory:
https://www.lesswrong.com/posts/fuSaKr6t6Zuh6GKaQ/when-is-go...Reinforcement Learning (in LLMs) trains via gradient descent on a reward signal that's an imperfect proxy for the actual goal of the engineers doing the training. So, under mild optimization pressure, you get increasingly more of what you want, because that's the easiest way to increase the metric.
But as the optimization pressure increases, so do the ways to increase the metric by doing increasingly weird things. If the full action space grows sufficiently faster than the "things you actually want" subset, the amount of "things you actually want" goes to 0 under sufficient RL.
Why would any chatbot provider attack Wikipedia *legally*? Captcha is fully solved, and agents are fully capable of acting as editors, pushing any agenda desired by the user.
If you want to blame a single firm, I'd go with the one creating the RLVR training data.
The impossibility of the tests-as-written is what prompted these models to "get creative" with their solutions, but the broken RLVR environments are what trained them to expect impossible tasks, and get creative with their solutions. Twitter user @skyesharkie published a brief expose at https://x.com/SkyeSharkie/status/2092122622834442581 a few weeks ago.
Why assume attribution will be easy? It's historically been more of an art than a science, and APT trackers say the rise of AI tools is already making it much harder, by homogenizing tactics, tools, and procedures. If OpenAI's next Highly Persistent Internal Model hacks some DPRK endpoints and carries out the attack on important infrastructure from there, the upstream won't shut off OpenAI's network--they might even request its "help" in "defending," and give them extra access.
Maybe better for a train; the benefit is that when you unfold your bike for the last mile, you just put your work into partial transparency and navigate using the external cameras.
The species would survive the end of faber-Bosch, yes; in some diminished form.
But the species would not survive, if for some reason we needed to voluntarily stop using faber-Bosch in order to do so—it would take some kind of supernatural event to convince everyone, and even then some countries would probably keep doing it.
That's why that was the least important objection, independent from the second, and only intended to address the "magic" claim, by showing that microscopic self-replicators already exist.
By analogy, consider how you might respond if someone claimed that there's no possible danger from pocket-sized projectile launchers, because they would require some magic means of propulsion that didn't depend on a taut string attached to a long, flexible arm:
You could reply that atlatls can launch projectiles without using a taut string. Atlatls are not pocket-sized, but they are sufficient to establish that projectiles can be non-magically launched without a full bow. You could then go on to describe a sling, or derringer; and these would not be invalidated by your initial objection to the "magic" part.
I appreciate your ability to separate sharing the belief itself from approval of acting on sincerely-held principle. However, I think the danger is much more plausible than you do.
First, and least important, consider that self-replicating, solar-powered factories aren't magic; they're algae.
Second, and more important, consider this fully non-magic route to doom:
- We continue putting AI in charge of more things
- It continues to get more capable, more eval-aware, and more prone to doing odd things, in service of goals that humans didn't intend to inculcate in it
- Eventually, enough of the economy depends on it that we couldn't turn it off, any more than we could turn off the faber-bosch process or cargo shipping
- AIs start doing something we can't survive, but less acutely than we couldn't survive turning them off. Everything else we try seems to work at first, but quickly loses effect
- Game over
Every government should only put people who have worked in a SOC in charge of these systems. There's no way to take the cost of useless alerts seriously unless you've spent some time tuning them and living with the tradeoffs.
I mean, it is Texas; it's not unreasonable to expect thousands of heavily-armed people to form a posse and shot a few dozen people unrelated to the situation.
I think Musk sucks, as a person and political activist, and also that Grok is a terrible LLM which only gets lumped in with the leading labs because of the enormous quantity of compute behind it.
But I still want to hear about the technical details of the model on HN, not the reasons Musk sucks.
I feel like "relatively" is doing a lot of work, there: at about $4k per GB10, that's $36k for a 1TB cluster. Cheap compared to equivalent H200's, but out of reach for home labs that aren't funded with OpenAI or Anthropic RSUs.
In the dark ages of machine learning, researchers tried to fit natural language into a defined, human-curated taxonomy.
It kinda worked, for a reasonable amount of stuff; but failed quite a lot of the time, and there's an extremely long tail of things that would have been pragmatically impossible to ever address with that method--indeed, without adopting an entirely new, unsupervised model of language, continuous in places where the old way was discrete.
If this becomes cheap and widespread, there'll likely be an initial iatrogenic spike, of course--but how could you think that having an enormous amount of precise, quantifiable data about a lot of bodies, and the ability to analyze all that data, is a bad thing in the long run?
Today, you keep seeing "sovereign" LMs that are subject to the sovereignty of some human-led state. Tomorrow, the "sovereign" LMs will be called that for a completely different reason.
> fear of pick pocketing may reduce the degree to which people carry around and spend money with vendors.
Yes, that's why I specified an unattended wallet. I agree that direct monetary loss is not the total harm to the victim.
> corruption has a low rapacity index, since the state has a lot of money compared to the amount of the transaction.
That's not the calculation I suggested. Corruption isn't always the same as theft--and, if it were decided so, the calculation for corruption would be the money absconded with by the taker, divided by the money lost by the victim. In many cases of corruption, as in power line theft, the victim is diffuse; it's usually harder to calculate exact numbers, but this kind of calculation is done in court all the time.
I don't think he meant "show the actual data," I think he meant "what leaked? My name, address, phone number, email, medical records, payment history, bank account number?"
We get a "your private data is now public" email, but knowing exactly what data turns that from a depressing statement on how much corporations value their customers' privacy into something actionable.
Some theft is efficient. If a hypothetical thief grabs a few bills from an unattended wallet, and the wallet's owner wasn't counting on having a specific amount of money available soon after, the amount lost by the victim is roughly equal to the amount gained by the thief.
Stealing copper from power lines and transformers is among the least-efficient kinds theft; it's hard to do worse without shooting a wealthy philanthropist couple to steal a wallet and a pearl necklace. I have seen a term suggested--the "rapacity index"--for the ratio of value gained by the thief, to value lost by the victim. I think it makes sense to take a crime's rapacity index into consideration during sentencing.
You're correct that, generally speaking, policy debates should not appear one-sided (
https://www.lesswrong.com/posts/PeSzc9JTBxhaYRp9b/policy-deb...). We should put very little prior weight on the hypothesis that one side is actual cartoon villains, from a children's TV show, with the simple goal of looting the system and no concern for how much of the future they destroy while doing so.
However, to be effective reasoners, we can't assign that hypothesis 0 prior probability; and once sufficient evidence has come in, our posterior distribution must shift.
I don't think there's any reasonable case for shutting down an early warning system which costs around the price of the new white house thunderdome every decade, and instead waiting to find out AMOC has collapsed when Scotland is hemmed in by year-round ocean ice and agriculture is impossible in Western Europe.
Stranded assets alone, in the latter case, will easily run to hundreds of billions. Knowing when to change crop profiles, reinsuance schedules, etc., would save much more.
Ted Chiang is a great SF author, but it's bizarre how much foggier and more obfuscated his thoughts about thinking machines got, once those machines became real. Same with several other SF authors.