In areas where you can easily recognise valid data (math, programming, gaming, to some extent also visual and mechanical), the synthetic data can be way broader. Using manmade data as input speeds up early stages of the process but is not really necessary. For later stages you run out of manmade data, and it’s quite easy to generate new one.

With writing/art it’s different because there you need and always will need human taste to judge. To some extent it can be overcome - you train models to be judges on what output is good in taste, but it can only go so far.

In principle you could have a training system that just gets hello world as the only program and develop all the algorithms and structures that we have now and more. In art, if you gave it a single cave drawing, it would either stick to cave drawings or - more likely - develop its own art that would be good according to internal taste, but non enjoyable to humans.

I think there is a bug misunderstanding if this in the art community. Most people there seem to assume AI needs human data to be trained. To create something that people like yes, to create something that works in math/programming/science/engineering - not really.