numeri

I think capitalization messed up the title here: It should be N^o(1)

> I’m happier to have more questions answered than for fewer answered but in an artisanal fashion.

Given that this is pure mathematics we're talking about, for the most part: Why?

Perhaps you are a mathematician and so these questions are relevant to you, but for most of us, these questions and their answers will not make any tangible difference in our lives or society.

You don't hire someone to "run math" in the context of all these discussions, which are about pure mathematics.

... thinking is how I feed my child. Telling me to have fun is such a tasteless and tactless suggestion

"And it and those robots will be owned by people who don't care if you starve or freeze to death."

And unlike those in Jamestown, who could just ignore the robots and continue working the land to survive, we will be forced to allow the robots to take over our roles.

Did you let it profile the code?

What do you mean by iterative chunking? There are Transformer variants that hierarchically build token- or word-level representations from bytes, then reverse it to decode, but they never really took off. Didn't help much compared to a good tokenizer.

yes, melatonin is unfortunately misused commonly, especially in the US, where it's not regulated for whatever reason. If you take 5 to 20 mg, it can act as an immediate sleep aid, by overwhelming your body with unnatural amounts, with a higher chance of side effects like nightmares. For reference, the body naturally produces about 30 mcg to signal itself to fall asleep [1].

If you actually want to shift your circadian rhythm, the optimal dose is about 300 to 500 mcg (the actual absorption rate varies wildly between manufacturers and delivery methods), taken 4 to 6 hours before your sleep onset (note: when you usually go to sleep, not when you want to go to sleep). [2]

[1] https://en.wikipedia.org/wiki/Melatonin

[2] https://www.circadiansleepdisorders.org/info/PRC.php

This is used quite often in SoTA quantization algorithms, which definitely count as production

I think this is the perfect example of how using Claude can trick you into thinking you've done something more impressive, when you're using it beyond your own understanding.

As far as I understand, you downloaded the structure of a neural network, ran it with essentially arbitrary weights, trained a classifier on the essentially arbitrary behavior, then simulated an approximation of that arbitrary behavior.

You could argue that there might be biases towards certain behaviors encoded in the connectivity, and I'm sure you'd be right, but your experiment is incapable of differentiating between those interesting behaviors and random noise. Especially because flies sorta act random anyway.

But I'm not claiming infinite harm with infinitesimal chances. I'm talking about finite harms with what seem to me to be non-zero chances. That makes it normal cost-benefit/risk analysis.

If you're trying to argue that the chances of a non-human source of greater than human intelligence having different goals than us is infinitesimally small, then I'd be interested in what arguments you have to back that up.

The problem isn't about what you think is the most likely scenario, the problem is the possibility of any of the many worst cases.

What does kowtow mean here? Say the other person is better than them?

Asking people to help you (within reason and without exhausting their resources), actually makes people like you more.

You have to be a minimum amount of likeable in the first place, though.

The models in these breaches have already been caught using proxies to get to servers outside of the approved list.

They've also hacked third party machines and used them to launch attacks on further services.

AI's learned their style through RLHF, with semi-focused, partially motivated humans giving feedback on short(ish)-form content.

For the most part, Modelese is the revealed preference of the average, not-heavily-invested human. This doesn't fully explain the Claudish coming from Opus 5 and Fable (this seems like it may be due to excessive RLVR or RLAI), but yeah.

It's got a lot of cheap writing tricks that make people think it's smart and helpful.

Fair enough :) I agree, and would boil down the article to "Assuming they are conscious will have horrible consequences for our society" instead.

I think that's quite plausible, but do think that if they're conscious, then it's our moral duty to accept those consequences and act accordingly (or stop creating conscious beings). The real trick is just knowing whether they are or not.

Stating loudly that something is obvious does not make it so.

Until we all know what consciousness is, this debate is going to continue going in circles.

This is unrelated to the article, maybe you replied to the wrong article?

This training technique does not relate to how persistent a model is, at all really. They sample more parallel attempts at hard problems, to increase their chances of having at least one success to learn from.

How would getting Chinese competition banned in the US prevent them from continuing to develop their LLMs?

Unless you're suggesting military action

That's such a shit parallel example that it borders on dishonest.

There are hundreds of incredibly strong scientific priors that would have to be disproven for the moon to contribute to the solution.

If a model was trained on this data, even if it was trained using methods that lead you to believe it unlikely to have learned details about the proof (e.g., maybe it was only used to train some kind of reward model, which played a minor role in the overall training and would thus be very unlikely to transfer details of a proof), you wouldn't have to disprove large swathes of known science to be wrong.

It makes me sad to think about. I would love to get into the new UI, but the immersion just won't come back. My muscle memory, hands firmly on the keyboard, is too persistent, and playing with the new UI feels like stumbling around and misclicking.

there are also people who have become aphantasiac after neurological damage, which seems like pretty cut and dry evidence against the qualia argument.

Quoting myself from a thread on aphantasia several months ago:

… there are plenty of scientific experiments that show actual differences between people who report aphantasia and those who don't, including different stress responses to frightening non-visual descriptions, different susceptibility to something called image priming, lower "cortical excitability in the primary visual cortex", and more: https://en.wikipedia.org/wiki/Aphantasia

So we know that at least the people who claim to see nothing act differently. Could it just be that people who act differently describe the sensation differently, you might ask?

No, because there are actual cases of acquired aphantasia after neurological damage. These people used to belong to the group that claimed to be able to imagine visual images, got sick, then sought medical help when they could no longer visualize. For me, at least, that's pretty cut and dry evidence that it's not just differing descriptions of the same (or similar) sensations.

But a senior engineer is only able to effectively delegate to interns because of years spent as that intern/a junior engineer.

If you're a junior engineer or an intern doing this, I think it might be harmful long-term.

How can a screwdriver do that?

If you're using AI to learn a new field (by which I assume you mean asking it what literature to read, asking questions when you don't understand something, etc.), you are accepting short-term speed in exchange for the long-term growth of research skills. Maybe that trade-off is worth it, if you don't need to master a topic long-term.

If you do need to master a topic, the slow and painful bits are the useful bits. Spending two hours hunting for an answer to something you don't understand increases your familiarity with the available resources, broadens your knowledge base, leads you to thinking of more questions about the subject and to understand your original question more deeply.

Shortcutting all of that with a quick answer from an LLM is not helpful long-term.

"pursuing advanced exploitation" when explicitly given a sandbox in a VM and a benchmark problem involving a cyber exploit very clearly excludes hacking third parties. I think writing out the event in a 3 point list like that is disingenuous.

This is basic alignment, not even a tricky or ambiguous case.

I do very much agree with your take on culpability/military parallels, though.

> new space is created

That seems to be the crux here. You think it will be, I (and a lot of other people) aren't sure it will. If new space for jobs are created, I am certain we'll be fine long term.

What do you think will happen if it isn't (I know you have strong evidence for and reasons to believe it will be, but just indulge the hypothetical)?