the root cause might be yocto which itself is very complex? xilinx just added its own layer on top when the board was made, and it stays un-updated for a few years, you got something initially working, and you need update it yourself after running its demos.

i use petalinux, not great not terribly bad either, I would say it's average.

I wouldn't be surprised if this were the case. IME you're better off either going from scratch, using gentoo, or using a different distro rootfs if they have appropriate binaries and just maintaining a custom kernel.