Absolutely, and seems like a feature that would already be in there if not for the majority of Firefox / Chrome funding coming from the advertisers.

IIRC it's already something you can do in Firefox with am extension, I vaguely recall using that to dump a video I wanted to watch offline.

There are two streams, video and audio are separate files. Thats the first hurdle stopping browser from presenting you with a simple "download source" button like on normal Video elements.