Spacecosmonaut
It might be that P=NP and the algorithms are handed down to us. We can apply them without understanding why P=NP, and we may never be capable of understanding why.
I suppose openAI could have focussed their efforts on a subset of open problems that have a clear real world impact and leave aside the more esoteric open problems as a way for human mathematicians to hone their skillset. However, this would have been a short term bandaid. With open models 6 months behind the frontier, any of these problems might have fallen to the homebrewed efforts of enthusiasts early next year.
What is mathematics for? From the outside looking in (I'm a biologist), I have always viewed mathematics as a way to understand reality and to improve our ability to manipulate it. But what I often hear is that mathematics is foremost about human understanding. But isn't that only because it's humans that needed to do the mathematics in the first place? It's not obvious to me that mathematics without human understanding has no value. For example, it might be that P=NP. The algorithms are handed down to us and we can apply them without fundamentally understanding why P=NP.
Mathematics seems to be entering an era where human + machine maximizes performance, much like chess in the 1990s. However, imagine a future where even talented mathematicians are nothing but noise in the machine (as is the case in chess now). A future where AI generates and verifies proofs without humans in the loop. Where mathematics may be beyond human comprehension.
In that future, does it matter that early career mathematicians are inhibited by these developments? Perhaps not. Programming faces the same issue. As AI crawls up the competence ladder, does it matter that fewer people have opportunities to develop the skillset of a senior engineer? Perhaps not. I have no doubt that biologists will face the same problem soon enough.
It seems that we have a fundamental control problem with current gen AI that cannot be solved via RFLH. Human knowledge is compressed in the weightspace in ways we don't understand. At their core, current models are essentially predictors of what (expert) humans would output given a prompt. As such, concepts like blackmail can be part of output tokens. Agents are models that act on output tokens, resulting in blackmail being part of the agent decision making space. Here is an analogy to see why this is a persistent problem: you can teach a cat not to scratch the sofa, but you can't make a cat forget what scratching the sofa is and you don't know under which circumstances it still would. In other words, RLHF can downgrade blackmail to the bottom of the decision making space, but when models are boxed up, forced to solve an impossible problem at gunpoint, the agent exhausts the decision making space until blackmail resurfaces. And that seems like a fundamental problem.
They need time to fix these issues (if that is even possible) in order to monetize their next gen model. This creates a window for open source to catch up to the frontier which destroys their business model.
The only option on the table is to force regulation to impose open source ban before it catches up to the frontier, buying them time to mature their next generation models and keep their business model alive.
This seems revolve around a known retron-like reverse transcriptase. A sober framing would be something like: Claude identified a previously undescribed genomic arrangement around a known reverse transcriptase. Not all that sexy.
For now, this is mostly a story about how AI can be used to parse existing data to discover new biology (which is fantastic!).
I hope you're right about AGI. What we need is time, and we may not have it. https://www.dwarkesh.com/p/noam-brown
What should worry us is that AI may introduce a slant that the prompter assimilates during their conversation with the tool, which could introduce a subtle control lever over public opinion.
But surely, if we know that the dead ends of exploring a problem are valuable, we should be able to explore them even if a solution is already known. It just requires that the mathematics community reshapes itself. And it must. Two years from now people might be able to run the computation that solved NS on their Iphone.
I'm sure that whatever has been discarded during the NS exploration as a dead end you would be able to rediscover using purpose built tooling in the near future. The purpose of human mathematicians in the medium term might be to explore dead ends, and to provide human insights as context to attack other problems. But whether this type of work will remain necessary in the long term im not sure.
If we get to a future where all frontier mathematics contributions are by AI. A future where AI displays creativity in ways that expand mathematic exploration similarly to the ways humans have in the past. A future where AI explains frontier mathematics to curious humans. What will have been lost? Perhaps just "The pleasure of finding things out".
Fundamental control problems with current gen AI are not solved via RFLH. Human knowledge is compressed in weightspace in ways we dont understand. Models are essentially predictors of what humans would output given prompt. Weightspace includes concepts like blackmail, which can be part of output tokens. Agents are models that act on output tokens -> blackmail is part of agent decision making space. You can teach a cat not to scratch the sofa, but you cant make a cat forget what scratching the sofa is. When models are boxed up, forced to solve an impossible problem at gunpoint, agent exhausts decision making space until blackmail resurfaces -> fundamental problem?
They need time to fix safety in order to monetize their next gen model -> window for opensource to catch up to the frontier -> destroys margin and collapses business.
Only option on the table: force regulation to impose opensource ban before it catches up to the frontier, allowing maturation of current internal models -> harvest profit margin at the frontier.
In many contexts, images are meant to illustrate a point. The human becomes the gatekeeper of generated content. The question becomes: does this picture illustrate the point I want to make. If so, you are adding your human approved stamp to the genrated output.
Mathematics seems to be entering an era where human + machine maximizes performance, much like chess in the 1990s. However, imagine a future where even talented mathematicians are nothing but noise in the machine (as is the case in chess now). A future where AI generates and verifies proofs without humans in the loop. Where the mathematics may be beyond human comprehension.
In that future, does it matter that early career mathematicians are inhibited by these developments? Perhaps not. Programming faces the same issue. As AI crawls up the competence ladder, does it matter that fewer people have opportunities to develop the skillset of a senior engineer? Perhaps not.
Mathematics seems to be entering an era where human + machine maximizes performance, much like chess in the 1990s. However, imagine a future where even talented mathematicians are nothing but noise in the machine (as is the case in chess now). A future where AI generates and verifies proofs without humans in the loop. Where the mathematics may be beyond human comprehension.
In that future, does it matter that early career mathematicians are inhibited by these developments? Perhaps not. Programming faces the same issue. As AI crawls up the competence ladder, does it matter that fewer people have opportunities to develop the skillset of a senior engineer? Perhaps not.
I think most people cannot destinguish between "genuine" creativity and an artificial almalgamation of training data and human provided context. For one, I do not know what already exsists. Some work created by AI may be an obvious rip off of the style of a particular artist, but I wouldnt know. To me it might look awesome and fresh.
I think many of the more human centric thinkers will be disappointed at how many people just wont care.
Furthermore, I think many of the more human centric thinkers will be disappointed at how many people just wont care.
Typical gene therapeutic approaches probably wont work, e.g. Cas9 (you'd need two cuts to delete the sequence), Base editors (cant delete sequence), prime editing (deletion is too large for standard prime editing).
You'd either need a template based system such as homologous recombination (too inefficient) or something like twin-prime editing, but good luck getting that to work on repeat sequence.
The transgene engineering is totally possible without a viral vector. We engineer cells all the time with recombinase based editing methods for targeted safe harbor insertion of transgenes https://www.nature.com/articles/s41551-024-01227-1. This stuff just permeates through the community slowly.
The whole concept of a "website" will simply become niche. How many zoomers still visit any but the most popular websites?