You don't need to do anything to pretend you are the source address. Whereas in TCP you need to guess some random seq number.
But if you're using encryption anyways, you're probably using some sort of "sign this challenge" message to verify the user instead, which would verify that that IP received the challenge anyways.
In what way is MITM easier in UDP than TCP? Assuming you use similar techniques to verify your connection, I can't see a difference.