Indeed, since the HTTP/1.1 RFC was issued in 1999 and the SRV RFC in 2000 there is no way it could have mentioned it.
If there are technical reasons why it can't work that's one thing. If some verbiage in the RFC is all that holds it back it's easy to fix: try it out, then issue a new RFC.
The whole RFC process embraces changes like this.
SHOULD NOT This phrase, or the phrase "NOT RECOMMENDED" mean that
there may exist valid reasons in particular circumstances when the
particular behavior is acceptable or even useful, but the full
implications should be understood and the case carefully weighed
before implementing any behavior described with this label.And an RFC is not a standard, and all life is meaningless? No, a SHOULD NOT should be taken seriously, not ignored.
The SRV RFC included that language for a reason, and probably had HTTP in mind when they wrote it, and therefore wanted HTTP clients not to use SRV records randomly. Since then, HTTP has has plenty of time to add SRV usage in its minor revisions, but has chosen not to. But now, with HTTP/2, a major revision, it is the perfect time to add this kind of thing.
RFCs are not like ISO standards. The name says it all: "request for comments".
RFCs reflect the ideal of "rough consensus and running code".
The SRV RFC was written by an author from Microsoft aimed at domain controller discovery. They probably didn't think of HTTP at all.
There's an expired draft RFC for SRV+HTTP[1].
You are right: SHOULD NOT should be taken seriously. But if you are (say) Mozilla, Microsoft or Google and writing a webbrowser you probably know enough to understand when you can break a RFC.
How do you think TCP-fast-start happened?
The SRV RFC was not by Microsoft. It has three authors, the other two being one Arnt Gulbrandsen from Troll Tech (of the Qt library fame) and Paul Vixie of the Internet Software Consortium, who is also the author of BIND. I actually am inclined to think that they did consider HTTP seriously.
(I wrote about this previously here on HN: https://news.ycombinator.com/item?id=7522208)