Then any source patch would fall outside of licensing.

It's clearly a derivative work, as described.

Some source patches include significant elements of what they apply to.

Some don't.

I think that argument would also imply that you don't need a copyright license to write a sequel to a book (assuming you don't quote the book, but the sequel includes characters / plots from the original book), and I think it's the case that a sequel is a derivative work. So by contradiction, it doesn't matter if the patch to the source includes a nontrivial portion of the source; it's still based on the source either way.

That's not the issue at all, it's still a derivative work, regardless of whether it contains much of the original.