Is your position that Javascript shouldn't be able to see click and drag events?

That would make a lot of browser games, tools (e.g. broswer image editors) and certain browser drag and drop functionality (e.g. a custom video or audio scrub bar) not work. They would require permission popups.