I use a custom URI combined with userjs(tempermonkey), mplayer and youtube-dl for same purpose.

First you add custom URI in windows, lets call it magnet3 (this should be a clue, it works just like torrent magnet links):

  [HKEY_CURRENT_USER\SOFTWARE\Classes\Magnet3]
   @="Magnet URI"
   "URL Protocol"=""
   "Content Type"="application/x-magnet"

  [HKEY_CURRENT_USER\SOFTWARE\Classes\Magnet3\shell]
  @="open"

  [HKEY_CURRENT_USER\SOFTWARE\Classes\Magnet3\shell\open]

  [HKEY_CURRENT_USER\SOFTWARE\Classes\Magnet3\shell\open\command]
  @="\"C:\\Users\\me\\Documents\\js\\youtube-dl mplayer.bat\" \"%1\""

then create your 'youtube-dl mplayer.bat' file (the %var:~9% nonsense removes first 9 characters from the passed link, '"magnet3:'):

  set var=%1
  for /f "delims=" %%A in ('d:\_learning\youtube-dl.exe -f best -g "%var:~9%') do start "" "C:\Program Files\smplayer\smplayer" "%%A"

or alternatively if your injected UserJS can extract direct link to media skip youtube-dl step:

  set var=%1
  start "" "C:\Program Files\smplayer\smplayer" "%var:~9%

now all you have to do is add userjs for the websites/services/players you want, for example I have a patched https://greasyfork.org/scripts/7096-convert-youtube-embeds-t... that replaces YT EMBEDs with a thumbnail linking to magnet3:link_to_yt, this is enough for starting mplayer on YT embeds.

Rube Goldberg sequence of hacks, but works perfectly.