AFAIK the PSGML mode is also the favourite one to use for XML editing.
nah, your best bet's definitely nXML - http://www.thaiopensource.com/nxml-mode/
...using http://www.thaiopensource.com/relaxng/trang.html for DTD -> RelaxNG conversion, I agree.
This seems like a failing of Emacs+PSGML rather than HTML5 lack of DTD.
Or rather why are you using an SGML tool to write a language not based on SGML? (from wikipedia "The HTML5 syntax is no longer based on SGML")