How do you want them to get better? Single core freq at the top end is the only advantage I believe Intel has at this point. AMD is cheaper, more IO, more cores at roughly equivalent speeds. If you are building a 10k gaming rig, I understand, but in general I don't see why you wouldn't go with AMD unless you need it to be have a better TDP. To that point, I expect Intel to continue to rule the laptop market.

Edit: clarified that I don't see Intel not ruling laptop market.

They could look to beat the i5-8400 in gaming perf. Don't really care though since gaming is an afterthought for the ryzen/threadripper plan to own the server market.

Gaming is the only space where intel is has an advantage over ryzen. It's better or as good in the rest of the workflows already

What's holding AMD back on TDP? I'm in the market for a new PC fairly soon - probably SFF desktop rather than laptop, but I'd still prefer to keep fan noise, heat and electricity usage down.

TDP is quite high on ryzen chips, and they auto overclock. The 2nd gen will go as high as it can with safe voltages, provided thermals and power are in check

I understand that it's high, I'm asking why. Are they not interested in the laptop market? Does Intel have some secret sauce they can't replicate?

Bigger transistors and more cores. https://youtu.be/ucMQermB9wQ describes this in more detail.

Likely. AMD uses Global Foundries to fab their CPUs. GF's fabs aren't as advanced as Intels, and ramping up clocks (thus more TDP) is the only way AMD can stay competitive on single thread performance.