Don't forget the amazing work being done at http://testthewebforward.org/ with the Web Platform Tests and your own work on the CSSWG tests! Beyond the specs, having a cross-browser set of tests that have especially good coverage on newer specs has been a huge leg-up for Servo!
It not only helps us ensure interop for those features that are tested, but (puts on former dev manager hat) it really helps quantify how many work items are left until a feature is closer to completion. I especially notice it when there are areas like CSS 2.1 where the tests are less comprehensive and it's harder to quantify what our progress/status is.
These tests are super useful. They don't just test for the spec, they also test for known behavior, which makes it possible to find spec bugs.
Plus they contain a very good framework which we can easily add our own tests into, instead of rewriting an HTTP testing server and whatnot.
I wonder if it's worthwhile to spend some time working on the 2.1 testsuite to try and improve coverage at some point. Probably? Unlikely to find so many bugs in mature implementations, hence why so much effort has been spent elsewhere.
For too long it was considered OK to have a spec with either no tests, or tests that focused on the wrong thing ("is it possible to implement this spec" vs "do implementations behave in compatible ways"). It is still considered too OK for browser vendors to write their own tests for a feature and not share them with the rest of the community. However, pressure from other platforms is changing attitudes here as it becomes increasingly apparent that without a stronger degree of interoperability, developers will vote with their feet and target platforms with fewer bugs to trip over.
1) There are seem to still be regressions in pretty fundamental stuff when new features are added. Usually caught before things reach "release" trains, but I see it in bug trackers all the time.
2) Other layout engine managers I've talked with who want to do a layout system rewrite are terrified of anything that doesn't look like a large-scale refactoring because of this issue. I've had the "how are you testing and measuring progress / how do you know you're done with layout?" conversation with most of our engine peers at this point :-)
Honestly, from what I see, the amount you do is relatively minuscule compared with what Opera was doing ten years ago, while Opera had far more marketshare which is normally the best way to avoid web compat issues in the first place.
Even things as simple as line endings are implemented wrong (in HTTP too), and that can cause security consequences as proxies end up reading headers differently than the user agents. I've seen this live on the public Internet.
SIP and the IETF to some extent encourages this with Postel's Law, telling implementors they should guess what the intention of the message is. We need less of VB's On Error Resume Next and more panic-abort type functionality. Look at this insane document: https://tools.ietf.org/html/rfc4475 "SIP Torture Tests". The authors gleefully come up with ridiculous yet legal permutations of messages that are allowed under their arcane rules. The fact this exists should send the opposite message: simplify your damn protocol. And this is only at the parsing level!
I know people say that HTML could never have had strictness because it'd have been too hard, but I don't buy that. One common issue was getting nesting wrong, like <b><i></b></i> or leaving unclosed tags. By removing the (silly, honestly) name out of the ending tag and just using </>, a whole class of errors is removed. Add in a browser that just fails to render and explains exactly why, and people would quickly not publish pages that are broken.
I don't think it's as clear-cut as that. I think the main thing is error handling must be defined: it doesn't matter whether it's panic-abort or whether it's defined how to deal with any stream of bytes. Don't let implementers choose what they should do.
RFC 7230, for example, has error handling sometimes as SHOULD return 400 Bad Request and sometimes as MUST return 400 Bad Request. But that only applies to servers and proxies parsing requests, which are really the relatively well-implemented part of HTTP. (The vast majority come from browsers, which are always syntactically valid; responses can come from arbitrary CGI scripts and you get all kinds of syntactic nonsense there.) Sadly, there's comparatively few normative requirements when it comes to parsing responses, even ones which are needed for backwards compatibility (at what point do you conclude you're just talking to an HTTP/0.9 server?).
So yeah, "implementation deviations" is usually the issue, but not really English ambiguity, at least with web specs. I put my pedant hat on when looking at proposed changes to web specs, and I believe the rest of the community does too :)
That said, while I certainly have a fondness for formal specifications, I also always end up feeling like they have a lot of downsides. They're often very verbose and hard to skim, and they're also another language that anyone implementing the specification has to learn prior to implementing it (or they can just wing it, but then your formal specification is worth nothing!).
The wisdom from old days was to use relatively, simple, formal languages that were easy to train people on and use. B, Z, and TLA had good results plus lasted over time. The other thing was to do English spec, formal spec, and code side-by-side with each team looking for inconsistencies. The developers usually picked up the basics of the technique rather quickly but at least one specialist was always necessary to make first use go smoothly.
So, that's still my recommendation. A subset of those, especially focusing on states and contracts, done in same fashion against English specs and clear code should take lower work than going all Isabelle, NuPRL, or PVS on a problem. Second application is also always easier. Reuse where possible helps as with code.
So, some difficulties and training but probably easier than some frameworks, etc IT people learn today.
From when I've considered trying to formalise parts of the platform before, some of the things that often get hit very quickly are the fact that most of the web platform is defined in terms of sequences of Unicode code points, and ultimately any formalisation is going to have to deal with that (be it some complex parser—like HTML—or some simple validation of it matching a regular grammar). It has always seemed to me that to get much advantage you need to be able to programmatically assert statements about it (including, most obviously, pre-conditions are satisfied at all call-sites), but once you've gone from 2^256 to 2^1114112 characters you end up with such massive state explosion that even many symbolic tools start to struggle quickly.
Now, maybe I've tried using the wrong tools, or maybe I'm structuring things wrong, but it'd be nice to have some way to do this.
It's a two sided coin: HTML now defines a lot of the basic web platform parts in enough detail that you can just implement the spec and have a web browser that works, whereas in the 90s (and most of the 00s) browsers spent huge amounts of resources on reverse-engineering each other. Pretty much the only way to be web compatible then was to start with something already mostly web compatible (consider the fact that Apple's first two or three years of work on WebKit were essentially spent pretty much just fixing web compat bugs!).