Even back in 2021, OpenAI worked on learned optimizers [1], ie neural networks that can update neural networks. Looks like this is now part of their larger model runs …

[1]: https://arxiv.org/abs/2106.00958