Unless you want to insmod things in your main kernel like a cowboy, I don't see why you'd need architectures to match. Cross compilation is the proper way (for some architectures it would be quite hard to find a machine capable of compiling the kernel before the heat death of the universe...)