In the case of PiFM, the clock generator is being operated out of spec by a wide margin - it is rated for operation up to 25 MHz, but the code operates it at 100Mhz.

That makes it do some weired things - and I suspect one of those is that when adjusting the clock frequency (by changing the divisors), sometimes it stops oscillating for many microseconds.

Not sure what impact that has in the frequency domain, but I can't imagine it's good...

Did you try hooking it up to a spectrum analyzer or a fast scope with FFT?

A PLL is one of the worst for phase noise. It will have a divider chain which will produce jitter and harmonics of the ref osc, and must absolutely rely on careful tuning of the AFC loop to null out these spurs. And as the filter will be digital rather than analog, there is no chance of optimizing it.

But that is before we get to the modulation. That again will be digital (rather than analog) so by definition will be chock full of jitter, especially if the FM is generated by stepping the clock frequency in finite steps. Not only will this generate large amounts of jitter, but it will generate spurs of the PWM clock over a wide range of frequencies.

I see, thanks!