I think you are missing the point of the main criticism. It is not about not wanting results in terms of proofs.

New theories and insights are typically created while working out proofs. If proofs now suddenly fall out of the sky (cause LLMs create them) then that work is not done which means the substrate on which new theories and questions and conjectures used to be grown disappears. It's in that sense that the math community (and thereby society as a whole) will lose something.

It's similar to how software engineering will need to find a solution to train their next generation. Current generations have all been through manual steps of designing things from scratch and writing them by hand. That's what allows your 10x engineers to understand whether what their LLM tools are doing is good and how to massage those tools to do the right thing. A junior engineer who has only ever used LLMs to write code and create architectures does not just not have that experience but also won't acquire it. You can't just say "we don't pay them to have fun and learn, we pay them to produce results". In the short term that is the case, but in the long term you as a company and we as a community will lose out.

I'm not saying don't use AI tooling. I'm saying that this is a hard problem which we yet to have to find solutions and approaches to. As a software community as well as as society in general.

"A junior engineer who has only ever used LLMs to write code and create architectures does not just not have that experience but also won't acquire it."

My ego tends to agree, that how can they be ever competent, if they have not endured the same hardships as I had crunching trough problems and getting allmost lost in the details.

But I rather suspect, they will turn out fine. I know LLMs are great for me to learn and I think the young generation will learn what they need to learn to get the job done.

How can they learn hard things if they have an infinite number of easy things to do? This is a middlebrow version of doomscrolling disease.

Most people have trouble not peeking at the answers. Look at Stack Exchange's long success.

Because keeping all the easy things coordinated and understanding the big picture is still a hard task yet unsolved by LLM's? But yeah, who knows what happens once that change. I assume even after the singularity, it still makes sense, that we train some people to know what is going on ..

How well do you think someone will understand fractions or trigonometry if they always punch their math homework into Wolfram alpha?

The increasing pervasiveness of technology in US education has not produced more capable graduates.

The only thing potentially stopping these models from also outputting new theories along the way is the goal they were given.

I have to assume OpenAI is only prompting to solve problems, presumably they could also prompt to not interesting new theories or paths of research found along the way as well.

OpenAI is doing problems because they know they can't do higher theory yet.

I asusme they're doing problems because its an easy way to turn $40m of someone else's money into a catchy news headline.

If a modern Gauss, Von Neumann, and Ramanujan appeared and started dropping proofs from the sky, would people be saying the same things? And if they could live forever, so they wouldn't need to train their replacements?

Who cares about them? I want Tao to stop proving all the interesting problems I was planning to work on.

Gauss and Euler, and also Ramanujan (results without proofs, which is a bit like unreadable Lean) did that for their lifetimes.

Yes, and they are revered as geniuses, which makes it clear that this is all sour grapes. And surely if people died, went to heaven, and were able to talk with God whenever they wanted, they wouldn't be upset that now they could know the answer to any mystery whenever they'd like; they'd appreciate that now they have someone to guide them! Or were they similarly upset when lecturers handed them already completed theory in school? There's already enough developed theory that people don't have the time to learn it all as it is.

Not exactly, because we would have cool people to inspire us and hang out with us.

But your argument is nonsensical because even if Gauss and von Neumann appeared, they wouldn't go into random fields and just prove things mechanically. They'd have to attend seminars, teach others, collaborate with others, and generally inspire others with their brilliance. It's the precise lack of this activity that makes AI in math so reprehensible.

Your argument encapsulates a contradiction because human mathematicians wouldn't be dropping proofs arbitrarily like AI is doing. They would do something completely different. Even the best of them.

Gauss was generally quite secretive and Ramanujan would famously tell people answers that he had received from divine inspiration, often with no ability to articulate how he knew. Von Neumann did just go into random fields and revolutionize them. If the three of them did come back from the dead and form a little powerhouse group that barely collaborated with the outside and just started publishing results for everyone else to try to keep up with, they'd no doubt still be considered geniuses.

Give it six months and models might be able to explain things better than any human. They can already collaborate perfectly well if you ask them to. e.g. there was a post here a couple months ago where Tao shared his ChatGPT logs[0].

If you're not inspired by the ability to talk to a superintelligent machine, and can't find what you'd want to know, that's a you problem.

[0] https://news.ycombinator.com/item?id=49010345

> Give it six months

Ah, the "six months till AGI" meme, but unironically :)

Also, before citing Terence Tao on LLMs maybe you should read what he has to say about it...

It's been doing all of the actual coding part of my job for the better part of a year, and you're commenting on a post about how it just released another round of math breakthroughs, besting a bunch of top humans. It can search the web and analyze documents it finds for me. It can do reverse engineering. It can analyze and create images.

Not sure what your definition of AGI is, but it clearly has superhuman performance on most knowledge work already. Do you think after already having demonstrated that it can solve top problems, that the final frontier it won't be able to cross is explaining its solutions to the experts that were researching those problems, and eventually to e.g. grad student or postdoc level practitioners as a lecture course/set of notes?

> It's been doing all of the actual coding part of my job for the better part of a year,

Makes me think of the Upton Sinclair quote:

"It is difficult to get a man to understand something, when his salary depends on his not understanding it."

Not sure I follow: if it weren't incredibly useful, my salary wouldn't depend on it since it wouldn't be doing my work for me, innit? So if I am dependent on it, that's just agreeing that it's now replaced programming per se as a professional activity. That quote applies to the people selling the tool (not me), not the people who it will replace (me), and in the meantime who are buying/using it.

I think the point is that people (software engineers included), will be biased to believe AI won’t replace them because believing otherwise would cost them their job (and therefore salary).

I’m not taking a side on that debate, just clarifying the relevancy of that quote.

"Can do the coding part of my job" is your definition of "almost AGI"?

I don't know a single taxi driver who 5 years ago would have claimed "my car can now park itself, FSD must be barely 6 months away".

No, but when Waymos were able to do the driving part, they probably should've seen the writing on the wall.

Isn't it curious that while LLMs are apparently on the precipice of "AGI" and possessing superhuman capabilities for knowledge work, the frontier labs are still hiring armies of humans as "Account Associates", "Account Directors", "AI Support Engineers", and "Applied AI Engineers" (https://openai.com/careers/search/). Since those roles are much more pedestrian than top math problems, one might expect the frontier labs to be at the forefront of automating those jobs using their world-beating models. Wouldn't that be a more direct flex of their economic value?

I think the crosswords framing is a little silly, but I have to wonder what comes when we use our technology to optimize the fun and interesting parts out of every job. There's only so many years of my life I can dedicate to back-and-forths with a chatbot. What if we advance our glorious civilization but our jobs just get more and more thoughtless and miserable?

I think that's where people hope some kind if UBI or "universal high income" will save the day. Just don't think too hard about how it would actually be paid for, or how we can all have high income when that's a relative measure and we're all given the same amount of table scraps.

"universal high income" is not when everyone has high income, it's when everyone who doesn't have a high income is excluded from the universe. There will be few high income people, robots those people own, and the rest of us will be undesirables/illegals/felons/noncitizens of Ms-Apple-Meta-Tesla-Google-topia, who for arbitrary reasons XYZ (they didn't accept the EULA!) don't deserve universal high income (i.e. most people here will fall into that category).

What you're describing could well be how it ends up, but that isn't the future described by universal high income.

Yours is more likely in my opinion though, mainly because universal high income is completely infeasible and shaky even at the level of definition.

Then work part time, and enjoy your higher wealth to have fun in free time. Don't demand to have your cake and eat it too.

What higher wealth?

I don't know about you, but my job has become a lot more fun ever since it's become a lot more back-and-forth with the robot. It does all the tedious things for me. It gathers data. It creates prototypes. It makes the mechanical code changes that I want. It allows me to talk with it for a design discussion, and then my design simply appears. I ask it for monitoring dashboards and they simply appear. It records what we talked about, which is something that I never do.

Largely I thought that this is what you do once you're established in math (or any field) anyway. You have some ideas, but the details are kind of too tedious for you to work out, so you give it to grad students/postdocs. Senior engineers have some ideas, but the details are tedious to work out, so you give them to junior engineers.

Now, obviously in the meantime, there's the question of how do we train the next generation? Or do we need to train the next generation? And maybe while we work that out the answer becomes more shadowing/apprenticeship instead of farming out easy tasks.

Sure, but part of me feels like we've got a good 5 years left of that fun until we're not needed...

I'm not confident in that statement, but I think it's well within the realm of possibility. Then again, so is needing humans to architect / steer for another 2 decades. We live in exciting times!

Unless AI companies are willing to take on liability for bad decisions the bot makes some human who can be promoted or fired will need to roughly understand risks and tradeoffs of what ai does

We have people who want to fill those positions because there are enough people who find it rewarding enough. Take away reasons why they would find it rewarding and you will have fewer theories and results that will advance our civilisation.

And yes, fun counts. Nobody said this had to be only a hardship.

Money doesn't work that way though. There are plenty of jobs people would like to get paid to do, that doesn't mean someone needs to psy them to do it.

I'm well aware that if at some point AI is good enough to replace me as a software engineer then I won't have a job. I don't expect a company to continue to pay me simply because I enjoy it if there are cheaper options out there.

Math is no different.

As long as that company doesn't expect me to continue in their employment if I stop enjoying it, then we understand each other.

Total compensation includes fun.

Oh sure, you're well within your right to leave a job when you're sick of it. Nothing wring with that.

So what? Why should my life be ruled by money alone? Why should my life not have a mix of intrisic and extrinsic motivational drives? Why should I respect a company like OpenAI that disrespects my mix of extrinsic and extrinsic drives when I am the customer?

Or am I just a product to them?

This is really the critical thing: the fun is the incentive. (Or at least the dominant incentive in math, historically.) As economists like to say, the overarching lesson in economics is that incentives matter. Reduce the incentives and participation will decrease.

Perhaps that won't matter if we enter an era where AI participants are the main participants who matter for discovery-level mathematics. But it would likely be what economists would see as a market failure if only a small oligopoly of AI participants, closely held behind closed doors, is able to fill that intellectual role.

We're going to have a very different perspective on purpose going forward with these results. This has crossed a rubicon where human output itself is going to be completely outclassed by machines and we will have to find meaning elsewhere in life.

Then you don't understand the process at all. You will get something back, you will get an immense amount back. But (almost always) not directly. A mathematician will not suddenly solve a theorem that will enable a cure for cancer or better solar panels or whatever. But working in mathematics will build the gradual understanding that will enable those practical breakthroughs to take place. It's also the most important part of how the people that create those technical breakthroughs will be trained.

This is the most cogent argument I have heard: depriving mathematicians the ability to solve these challenging yet low ROI problems now prevents them from building the foundational knowledge needed to later solve something truly groundbreaking.

Whereas for an LLM, it's just a way to ~waste~ spend tokens.

Unless you think the future doesn't need mathematicians, we ought to respect their requests.