I agree with this. I often write up notes in Org mode, but, after trying for a while to have all of my 'sources' in Org, I also came to the conclusion that for actual papers, it's far better just to write them directly in LaTeX (other than initial notes).