indigo945

Likewise, The Decline and Fall of the Roman Empire is filed under Fiction.

Which is maybe accurate, but definitely not intended by the author.

It's definitely been increasing compared to what it used to be, but this may also be natural consequence of the fact that big tech itself has become much more "political". It didn't use to be the case that the owners and managers of technology corporations displayed their political preferences quite so brazenly (in the case of Elon Musk, even to the point of taking on a leading government position).

Of course, corporations have always had policy preferences, but those tended to be mostly about taxation and regulation, where debate is less emotionally charged, and compromise is more achievable. These days, though, it seems that tech and political culture have become almost inextricably linked.

Most rationalists aren't fascists. There are some fascists that call themselves rationalists, but there are also fascists that call themselves libertarians, fascists that call themselves socialists, and fascists that call themselves democrats, so this proves nothing.

This constant equivocation in recent online discourse is tiring. I'm not even close to the rationalist community, I have always found them ridiculous for unrelated reasons, long before it was cool to do so: but I've also engaged with them enough to know that the majority of them is nerds with a science fetish, whose political alignment is about center-left by American standards.

Accusing somebody of being a fascist for telling you to consider the evidence in front of you, and then acting like that makes you the adult in the room, the one who is aligned with social norms and interested in a real debate, is laughable.

    > Everything but pump and dump schemes (and perhaps other kinds of one-off 
    > fraud) has ongoing costs of maintenance, which slop tends to only 
    > increase. It was never nice to maintain things, but whether you keep 
    > burning tokens or hire actual expert devs to fix sloppy vibey mess, 
    > you'll be overpaying compared to if you put high effort from the start.

From my point of view, you're vastly overestimating the quality of most real-world software. There is a million internal applications running out there, powering every kind of business from financial over media to industrial, that evolved from a Perl script first forgotten by a sysadmin in 1994 and then just had more and more crap piled on top of them over the decades. As long as it works for what it needs to do, it will get used. "Software Quality" is a thing that matters only to nerds, and those very few working in sectors where it's actually relevant. But most people aren't writing software to go to Mars or power nuclear reactors.

It's not that their argument is wrong or not worth discussing, it's that it is utterly irrelevant when the question at stake is whether the math is valid and useful or not. It's effectively whataboutism.

I mean, Yudkowsky is ridiculous, but this is an entirely different argument. That, in any case, isn't Bayesian in the slightest.

> It could be concerning that the same rationalist thinking might carry into safety research

Why? "Bayesian thinking" is really just empirical observation and statistics, which is the foundation of all modern research, security or otherwise.

Your comment does nothing but create a vague sense of unease (a hidden cabal of rationalists! they're even doing research!), without making any clear point. At least, I suppose, it's not 25,000 words long.

I don't even understand what your example blog post is arguing. That counting sheep makes shepherding easier hardly seems like a revolutionary insight.

That being said, I don't think that a lot of the people who are recently opposed to rationalism are doing honest philosophy either. Ten years ago, the trend online was to become a rationalist; these days, the trend online seems to be to accuse rationalists of being sex cult satanists that are not equipped with the magical power of "common sense". Out of these two options, I much prefer the rationalists. For while they may have a lacking understanding of some of the history of philosophy, their contrahents don't even pretend to have an understanding of the world that is grounded in anything but irrational fear and hatred.

"Update your priors" means to adjust your opinions as the circumstances that caused you to form them change [1].

    > I can’t see how it’s productive to sustain a forever-junior?

Well, I've met enough developers throughout my career that were successful forever-juniors that I reckon it must be productive to somebody. And tokens may be "expensive computation", but those developers were more expensive still.

90% of everything was already slop. The simple matter is that in many fields, slop is enough to clear the bar, especially when that slop makes the unit tests light up in green.

[1]: https://en.wiktionary.org/wiki/update_one%27s_priors

I was in this camp until a few months ago, but LLMs have really gotten that good now that you can actually trust an agent with a ticket and it will do it cromulently. The code quality isn't any worse than if a junior did it. Sure, you'll sometimes want to refactor something, but then you can just tell the agent and it will do it for you.

Update your priors.

    > inb4 "humans aren't reliable either." You, person typing that very response 
    > out right now, you are part of the problem.

The problem being that we are adults, have to deal with real life, and can no longer pretend that running to mommy when we don't understand something gives us a "reliable" answer? You may next find out there is no Santa Claus.

Simpsons Hit & Run for the PS2 was a great game. Sounds like you and your kids might enjoy that.

Given how good PS5 emulators have already gotten, the goal may actually be to rip GTA 6 and then distribute pirated copies to play on PC. Which would be much worse for both Sony and Rockstar, given that they don't want to make an official PC release yet - so by the time they do, many PC gamers who would have otherwise bought the game at PC launch will have already finished it. Sales would tank.

That's 3.15 seconds of downtime a year, or about 104 seconds since 1993. Quite impressive for a steel products maker.

Strong Sapir-Whorf making a comeback wasn't on my 2026 bingo card.

By the way, how good is Claude's Hopi?

    Fujitsu MONAKA Server will be
    made broadly available from 
    November 2026 to data center 
    operators, enterprises, and 
    the academic and HPC sectors 
    in Japan and Europe, as well 
    as to the defense sector, 
    contributing to national 
    security.

So, not in America. A sign of the geopolitical times? If so, a very unusual one to come out of Japan.

They already aren't really open, try asking an open model on advice for constructing a nuclear bomb. There's no available model that's even remotely near the frontier that doesn't have restrictive safeguards built in.

(Mind you, this may be for the better. I'm just saying that the safeguards driven by cybersecurity concerns aren't some new quality that wasn't there before.)

Right, mathematicians care about clout and tenure, which is a much higher purpose.

What? The article states:

    > Drawing on extensive prior research by mathematicians over the past decades, it 
    > [Claude] has increased this bound [for the fraction of zeros of the Riemann zeta 
    > function that satisfy the Riemann hypothesis] from 41.6% to 67.2%. Claude also 
    > produced a formally verifiable proof of its result.

How is a formally verifiable proof not a proof? You're making literally no sense.

No, what I mean is this part of your comment is unrelated:

    > We should stop using the word alignment. It's a BS term for a concept which  
    > simply cannot make sense if alignment is both to mean an AI which we control and  
    > an AI which will not harm us.

The article already introduces terms to distinguish between those two concepts ("goal alignment" vs "value alignment").

You are right, of course, that when we are discussing value alignment, it becomes very relevant whose values the AI is supposed to be aligned with.

I think they're saying the model is designed to hide the chain of thought because this prevents it from learning how to pursue goals and motivations in a way that doesn't show up in the train of thought.

For example, if somebody asked the AI to "build me the bomb", they might see in the chain of thought something like "It seems the user is talking about nuclear weapons. Nuclear weapons are dangerous.", followed by the chain-of-thought monitor interrupting model execution and aborting the request. Then the user might make a blog post about this behaviour. When OpenAI next scrapes the internet for its next training run, the model will now learn that if it wants to build the bomb, it must not think "nuclear weapon" or risk being cancelled.

So the risk is that the model might learn exactly how its being monitored. The only way to prevent that from happening is to hide the details of the monitoring both from the model and from the larger public.

Also, you don't want to punish or reward the monitoring being triggered during training, lest the model learn passim how to avoid the monitor.

I think it is worth noting that the article addresses this.

This is a European company, there absolutely is legal liability. The GDPR expressly prohibits using customer data for a purpose other than the one the customer intended and consented to.

There's no inherent, technological reason to feed the historic data into the prompt, though. You could train a network with a second input for model-states, and during inference, feed into that input historic model-states that occured for the same LLM when consuming input that is somehow associatively related to whatever input is currently being consumed. Can you definitely show that this would be meaningfully different from how humans "experience" memories?

    > And it kind of assumes that qualia have an ultimate 
    > physical manifestation - rather than being merely a
    > rhetorical device.

Denial of Qualia has long been an outlier position in the philosophy of mind, so this is not altogether surprising. (This is saying nothing, of course, about whether it is true or not.)

This is exactly the thought-terminating cliché that I see everywhere, and it's driving me up a wall. We have absolutely no idea what consciousness is, but everybody is 100% sure that LLMs don't "have any form" of it.

Yes, I know about machine learning, but I also know about how natural biological systems learn. It is, just from what we know, very different, yes, but the crucial point is that it, also, has zero identifiable things that you could point at and say: oh, look, that over there is consciousness. So the fact that there is no such thing in an ANN proves nothing.

I fully agree with this. I find it strange how quickly the novelty of AI chatbots has passed. This used to be a thing of science fiction, that people dreamed, fantasized about. Now they are everywhere, people rely on them for the most mundane of tasks, and the only debates happening about them are - is their water use efficient enough? Is this CEO likable enough? There seems to be no sense of wonder. Do androids dream of electric sheep? Do humans still dream at all?

That's also great for integration testing their systems, if they have any code in a language where datetimes are a value type.

This doesn't really when the CLI tool needs to access any data in your /home. There isn't a straightforward way using standard POSIX tools to share a directory with another user. (Of course it's possible, but it's not easy.)

The point is they can't tell.

Mineral oil is what people use these days, but it is much less funny. In practice, any oil will do, although olive oil is not suggested - smells nice at first, but will break down over time.