> The AI is itself a program. If the AI can write programs, it can write a better version of itself.

Not necessarily (or even likely). There's a huge difference between being able to write "programs" (for example the ones described here are ~5 lines) and being able to write a large and very complex piece of software, which such an AI would undoubtedly be.

There's also the question of what constitutes "better." Improvements in e.g. speed or memory efficiency (hard enough as it is, but conceivable for an AI) are a quite different thing than improvements in capabilities or understanding.

> There's a huge difference between being able to write "programs" (for example the ones described here are ~5 lines) and being able to write a large and very complex piece of software, which such an AI would undoubtedly be.

That's true. real-life programs that programmers get paid to write are typically more complex than 5 lines. Any AI good enough to replace real programmers is going to be very close to good enough to rewrite itself.

This makes the assumption that every program it writes is somehow guaranteed to be better than the original. Why would it not run into the Multiplicity problem, where each subsequent copy is a bit dumber than the original?

I would think it would be much more likely follow a normal evolutionary model - improve itself to a local maxima, then just stall while the random mutations fail to make the huge leap required to hit the next maxima.

We can't even properly model the human mind in computers yet - not enough processing power - so it follows that an AI which is "better" than humans would require even more processing power than that required to ape humans.

The difference here is the "intelligence" of an AI can be described as a loss function, so if an AI can build and then train another AI which can achieve a better overall score from many examples in its loss function, then that AI, with perhaps a deep neural network will start to "understand" or the components that make a successful neural network. Thus it could use intelligence rather than random mutations to evolve.

I think things like psychology/psychiatry, social work, child protective services, and other jobs that rely on a lot of interpersonal communication will be around longer than development; at least at any scale worth talking about.