Because the other option is that AI does what you say. And if you think about it (and you are a mathematitian and not an engineer) it means that AI is able to read our minds and know our desires. And at that point who cares about existence anymore?

Sorry, why would it need to read our minds exactly?

"Choosing which terrain is worth exploring" is more clear if you have an objective function (if you are doing engineering). But mathematics is closer to art in that regard. What is a terrain worth exploring in art?

Sometimes I wonder if "art" is also just some form of pattern matching on a very meta level.

In some sense, any art can’t be random in order to be distinguishable from noise

Art is the ability to transform one experience into another. That's a generalization only RSI can hopefully come up with.

>What is a terrain worth exploring in art?

The joys of parenthood. The excitement of new love. The tragedy of aging. The rise and fall of a great family. Jealousy, vengeance, embarrassment, righteousness, venality. You obviously couldn’t create an all encompassing map of the terrain, but human art has a lot of recurring themes that are clearly “worth exploring” because they’ve consistently been explored in the past. I think the same kind of repetition probably also exists to some degree in other areas of human endeavor. An artificial intelligence could probably suss out a lot of those themes and produce valuable (if not maximally valuable) work without being a mind reader.

Im not saying it cant do those things. The mathematical equivalent of that are the hundred proofs that OpenAI dumped. And probably we still have a few decades of AI exploiting low hanging fruit in Maths and Art.

I was refering to going beyond and deciding what to explore. There are infinite paths in mathematics and a lot of ways you can describe and denote a particular theory. And the choice of notation and exposition is an art in some way.

You could probably say that "no, but this is done by minimizing memory usage in human brains so that its understandable by humans, maximizing compressability of previous ideas, and a link with some utility in the real world, etc"

But that just one example of loss function. There are many possible ones.

Do i think that there is a universal loss function that captures it all? Probably yes, in the limit, on the infinite energy regime. Are we in that case in practice? Probably not.

I tend to think that AI will fully replace us the same moment we are able to master prime factorization with our time and energy constraints.

Just because they could, doesn't mean anyone would bother having the AI agents do so.

You don't think OpenAI has any interest in having agents identify new interesting research terrain? They just want to solve existing important millennium problems and then they're done?

Yes? I expect their motivation for devoting resources to solve these problems will dramatically drop as soon as they IPO. Maths is just a convenient vanity metric for OpenAI to demonstrate progress, but long term progress in mathematics has no commercial value to them.

Of course, general improvements may be sufficient to continue to drive math improvements even without specific attention - we’ll see.

Not a chance. They'll go for room temperature superconductors and other stuff that is currently in the quite possible but not yet proven and patent the hell out of it and charge rent to the rest of humanity.

Why would that necessarily be accessible by thinking only? Or do you see them running or directing large laboratories of various types?

Simulation to the point that the accuracy is large enough to extrapolate to real materials. You can do a lot of stuff when you have access to a million GPUs that would otherwise be much harder.

I have my doubts there, but we will see.

A lot of materials science is already done through simulation, same with a large chunk of chemistry and biology. That fraction will only increase over time.

https://en.wikipedia.org/wiki/Computational_chemistry

https://en.wikipedia.org/wiki/Computational_biology

https://en.wikipedia.org/wiki/Computational_materials_scienc...

These fields are all going through a tremendous acceleration right now.

I am aware of advances, but I think a large chunk is a stretch in various ways. We'll just see what happens - simulating even a glass of water fully is tough.

That depends on the granularity of your simulation. For most - not all - purposes a glass of water can be simulated coarsely or by simulating a far smaller amount with greater fidelity and then to extrapolate.

For useful materials science the numbers are still large but far lower than the numbers involved in 'a glass of water'.

I really depends on what is needed. If you know that something quite "coarse" is enough to create/uncover room temperature superconductors with high critical current - fair enough. Again, we'll see what happens.

With the kind of compute they have at their disposal 'quite coarse' may actually be 'impressively detailed' and that is why I think they have an edge on this kind of stuff. It's like being able to borrow a trillion and then to go play research lab, it would be highly surprising if they weren't able to monetize such a head start.

AI systems will probably be able to, if not direct labs themselves, cut down a lot of the search space to only the most promising candidates. That’s basically what Isomorphic Labs is doing for candidate drug research. It doesn’t seem crazy that the same could be true for material sciences.

Maybe. Are we seeing any indication of increased success rates in stage 3 trials already for "AI drug candidates"?

Could also be that some things are not possible in some ways. I wonder if there is some general theory on "shortcuts".

Stage 1, yes, stage 2, it's a wash (for now at least), which does not promise much for stage 3, but it is still a bit early for that.

If they did that and were able to find someone to pay them, that'd be a pretty good contribution, no?

I think OpenAI's limitations are quickly verifiable results. 10,000 agents running in parallel aren't exactly talking to each other. They have a goal and one eventually hits the goal which can be automatically confirmed. This is why math is currently attacked, lean is easy to confirm A = B.

They aren't gonna create room temperature superconductors because they'll need a lab that can quickly, and verifiably, test superconductors at 10,000 per minute and that's just a physical limitation no matter how many datacenters you throw at it.

> I expect their motivation for devoting resources to solve these problems will dramatically drop as soon as they IPO. Maths is just a convenient vanity metric for OpenAI to demonstrate progress, but long term progress in mathematics has no commercial value to them.

And why would IPO remove incentive that's present pre-IPO? Somehow math progress will influence price at IPO but won't influence stock price post-IPO? Or will they just stop caring about stock price once they are public. None of these options makes any sense.

No, post-IPO they will have to (at some point) demonstrate real commercial value. This just gets them to a stage to cash out insiders and (less importantly) get some additional capital to extend their runway.

That's the playbook, right - keep interest/investment flowing until you can figure out the actual commercial value/model. Until then, do whatever you need to keep the hype as high as possible and maximise your runway.

Wouldn’t they have to do that even if they didn’t IPO? Also arguably after IPO hype is even more important - look at tesla valuation.

Eh, I'm sorry, but math being a vanity metric is not at all as factual as a take as you present it as. Maths are just a convenient target to improve their models through RLVR, using verifiable endpoints (ie. lean proofs) to generate novel training data. The publications and vanity stuff is just a side effect of their primary motivation: making the model better.

Because if you assume there is going to be an AI-god that can do everything better than us, there's nothing to really do. Like really, what do you want these people to write? "Well, math for humans is over as well as all intellectual activity, better just jerk off watching AI-generated hentai all day until I die"

Just because the truth is inconvenient doesn't make it false.

I didn't say it was impossible, I said if it were true there would be nothing to do and thus no point in writing about that potentiality. It's sort of like writing about what mathematics should do if there was a thermonuclear war that wiped out all of civilisation.

Going about your job/life despite a looming threat is a different level of resignation than telling others not to worry and giving them a false sense of hope not because you truly understand the danger to be fake but because "what else am I to tell them? the truth is too horrible to bear" isn't actually doing anyone a kindness: it is going out of your way to make the jobs of those who would tell you the raw truth harder.

If you go to the doctor and they understand there there is a chance you are going to die but, when you come to them for advice on your pain, they look you straight in the eye and insist that you will be fine if you just eat some more fiber, they rob you and your family of a not only a chance to prepare and grieve but the motivation to fight the inevitable.

To your analogy, if you think we might be on the brink of a war that might massively effect your life and so you go to the wise guru who assures you you will be fine if you just duck and cover in the basement, they are, frankly, part of the problem, as they are making the war sound "not that bad", when if you understood the threat of thermonuclear war you might be protesting in the streets.

I don't think you understand the conversation that these mathematicians are having. They are debating what to do as mathematicians with regards to math; not how society should handle a hypothetical AI god. I also don't think they necessarily believe in the AI god the way that you do.

It's not the truth until it's proven to be true though. It's a possibility, however likely or unlikely.

The alternative is sitting around and doing nothing and waiting for this God-like AI to come into existence, but you'd look pretty silly if you threw away years of your life only for that not to actually happen.

Putting the fear of looking silly ahead of fear of human extinction or other nefarious consequences that could happen with that God-AI is way sillier.

Look, if we were worried about human extinction, the frontier-labs should simply stop. They won't though.

If this scenario plays out in full it will be a bit worse than 'inconvenient' and I hope we will be able to prevent it.

Ive noticed that too, I don’t know why people talk like we need to figure this out right now

Like we can even know what the future of math is this early

It’s not like you can just declare that up front, it’s something that happens naturally when people spend time using the tools

Right now they have to figure out some kind of story to tell the grad students. Even if it will be subject to change they have to have something.