Not asking for a normative judgement or a positive one, I asked about what you thought would be the logical progression (under the assumption that humanity would like to continue to evolve and become more prosperous and becoming more intelligent or building more intelligent tools seems and obvious way to help enable that)

Could be more than one, was just trying to give some large buckets, that's why I added D and asked for people to add their thoughts.

Yah, but what is that?

Are you talking about an IQ score? Ability to do the tasks that ML is not currently good at isn't a great measure (a few years ago, Go or creating high quality paintings would have qualified).

E.g. one example of "D" is "rigorously studying and improving human education". But there's infinite clarification on what the problem you're trying to solve before we agree what improvement is:

* Mean, median, or peak outcomes?

* Performance on a given curriculum, or on a given abstract class of problems? Does creativity count? Is an education system that improves various kinds of quantitative metrics by 20% but results in a 5% less creative and flexible population a "win" or not?

Pretty much all the same arguments apply no matter which approach you consider.

> Not asking for a normative judgement or a positive one,

Well, normative considerations are important. If experimenting with brain-computer interfaces for quick advancement harms humans, is it a "good path"? How do we compare these paths? Quickness, magnitude of results, "best" resulting society, least harms, some kind of cost measure?

So often this blows up when anything related to IQ is brought up, a profound generational trauma of american intellectualism gets a small scene to replay its discourse in suffocating detail.

Maybe this is as close to the root cause as we can publicly get? Our insane tendency to avoid productive discussion and stifle true creative activity, all in favor of hollow wordplay (quasi-statistical if the situation requires, not less hollow). Our education - a glorified daycare with rote-learning for line-workers, of which we somehow expect intelligence gains if only we could spend more time instilling it (going against all solid evidence to the contrary, because the doctrine of never having enough education has to be preached).

There is a word for this, and it is a neologism: https://roonscape.substack.com/p/a-song-of-shapes-and-words

> a glorified daycare with rote-learning for line-workers, of which we somehow expect intelligence gains if only we could spend more time instilling it

Drilling tons of math in rigorous ways for excessive amounts of time clearly improves various kinds of outcome measures. It also tends to cause other harms.

If we want to improve intelligence, first we need to agree what intelligence is. Then we can talk about what measures are appropriate or ethical to use to try and improve it.

Even if we just take a somewhat stupid measure, and say "all we care about is a hybrid of IQ and SAT-I scores at age 16"-- there's still the question of whether we care more about moving the top performance upwards or the entire distribution.

If we don't agree what intelligence is, we can't answer questions like this. If we use a traditional measure like IQ as a measure of G factor-- I don't think brain-computer interfaces are going to swing things much and they may even make it worse. (But they could also allow explosions in human productivity or quality of life that are not measured on that scale).

e.g. Smartphones have not made us smarter, but they may sometimes allow us to do smarter and better things.

First, it's hard to rigorously reason about and convert to a quantitative metric.

Second, it would seem to imply that many geniuses in the math and sciences are not very intelligent.

Figuring out what we really mean is really hard.

An IQ score is a measure of intelligence, it's not intelligence itself. Just as the temperature you read off a thermometer is only a measure of the heat in the room. The thermometer tells you very little about the position and momentum of the gas particles in the room, only that higher temperatures correlate with more momentum.

The value of IQ scores for measuring human intelligence is that they correlate with all sorts of life outcomes. The hypothesis behind the research is that both the life outcomes and the IQ scores are caused by a single factor, known as g, for general intelligence.

Obviously, if we train some model on IQ tests we can probably get a machine to ridiculously high IQ scores. Does that mean we have created intelligence? No. The machine is not going to show any of the life outcomes we expect from a human with high IQ. It's not going to do anything! That's why none of these machines are intelligent.

The point is, what's "improving intelligence"? It's something that we have a hard time quantifying on a linear, agreed axis. As you point out, IQ scores are a somewhat flawed measure of g, which itself is a somewhat handwavy observation of a underlying factor that is a strong correlate to human performance in a broad set of areas.

And even if you could put that human performance in a linear measure, there's still a question of how you value different changes in that distribution. Which is better-- raising everyone by .1 standard deviations, or boosting a few people near the top by 1 standard deviation?