News for hackers. Stuff that matters.
Re:Servo Nightly Builds Available by pilif on Friday July 01, 2016 @05:02AM (#12014709)
It's amazing to see actual effort taking place to build a new browser engine from scratch.

I would say that since the late 90ies, there hasn't been a single new engine being made. Everything we have now is a development based on KHTML, Gecko, Presto or Trident.

By now, HTML and the standards surrounding it have become so big that starting from a blank slate nowadays is next to impossible. The existing engines have it much easier as they are allowed to grow with the spec, but a new engine has to do everything from scratch.

This makes it even more impressive to see how far Servo is already (yes. I know - there's probably some overlap with some existing Mozilla code, but still).

Huge congrats to everyone involved. We need unafraid visionaries like you people to actually move the web forward.

Re:Servo Nightly Builds Available by metajack on Friday July 01, 2016 @10:41AM (#12016379)
> By now, HTML and the standards surrounding it have become so big that starting from a blank slate nowadays is next to impossible.

One important mitigating factor is that newer standards often generalize older ones. For example, originally all the list elements had special case handling (and still might in other engines), but now it's possible to do all that with the user agent style sheet with new additions to CSS that generalized these things.

It saves an enormous amount of work to implement the web at a particular point in time as opposed to starting at the beginning and keeping up with the changes until that same point in time. Not only that, but all the failed branches that were already explored mean that a new engine avoids quite a bit of technical debt.

It's still quite hard. Here's a nice example of a layout bug on HN we thought would be simple to fix for the release, but turned ugly fast: https://github.com/servo/servo/issues/11821

Re:Servo Nightly Builds Available by nickpsecurity on Friday July 01, 2016 @11:21AM (#12016796)
This has been interesting test. Thanks for your suggestion on the -c option for "can't load servo." That worked. Filed a bug on HN login where tab autosubmits form instead of changes textbox. I can neither see the text I'm typing nor scroll down on HN. I'll check to see if open issues on that then file them if there aren't.
Re:Servo Nightly Builds Available by gsnedders on Friday July 01, 2016 @05:47AM (#12014831)
> By now, HTML and the standards surrounding it have become so big that starting from a blank slate nowadays is next to impossible.

It's a two sided coin: HTML now defines a lot of the basic web platform parts in enough detail that you can just implement the spec and have a web browser that works, whereas in the 90s (and most of the 00s) browsers spent huge amounts of resources on reverse-engineering each other. Pretty much the only way to be web compatible then was to start with something already mostly web compatible (consider the fact that Apple's first two or three years of work on WebKit were essentially spent pretty much just fixing web compat bugs!).

Re:Servo Nightly Builds Available by larsberg on Friday July 01, 2016 @08:25AM (#12015427)
> you can just implement the spec and have a web browser that works

Don't forget the amazing work being done at http://testthewebforward.org/ with the Web Platform Tests and your own work on the CSSWG tests! Beyond the specs, having a cross-browser set of tests that have especially good coverage on newer specs has been a huge leg-up for Servo!

It not only helps us ensure interop for those features that are tested, but (puts on former dev manager hat) it really helps quantify how many work items are left until a feature is closer to completion. I especially notice it when there are areas like CSS 2.1 where the tests are less comprehensive and it's harder to quantify what our progress/status is.

Re:Servo Nightly Builds Available by Manishearth on Friday July 01, 2016 @08:46AM (#12015531)
So much this.

These tests are super useful. They don't just test for the spec, they also test for known behavior, which makes it possible to find spec bugs.

Plus they contain a very good framework which we can easily add our own tests into, instead of rewriting an HTTP testing server and whatnot.

Re:Servo Nightly Builds Available by gsnedders on Friday July 01, 2016 @09:03AM (#12015646)
To be fair, having tests somewhat relies on having a spec. OK, so in theory we could've had a cross-browser set of tests with some requirement along the lines of "this web page relies on this behaviour", but ultimately you'd still end up with contradictory tests. :)

I wonder if it's worthwhile to spend some time working on the 2.1 testsuite to try and improve coverage at some point. Probably? Unlikely to find so many bugs in mature implementations, hence why so much effort has been spent elsewhere.

Re:Servo Nightly Builds Available by jgraham on Friday July 01, 2016 @10:46AM (#12016416)
<vested interest hat on>

For too long it was considered OK to have a spec with either no tests, or tests that focused on the wrong thing ("is it possible to implement this spec" vs "do implementations behave in compatible ways"). It is still considered too OK for browser vendors to write their own tests for a feature and not share them with the rest of the community. However, pressure from other platforms is changing attitudes here as it becomes increasingly apparent that without a stronger degree of interoperability, developers will vote with their feet and target platforms with fewer bugs to trip over.

Re:Servo Nightly Builds Available by larsberg on Friday July 01, 2016 @10:51AM (#12016469)
I think it's worthwhile! Lack of a comprehensive test suite means:

1) There are seem to still be regressions in pretty fundamental stuff when new features are added. Usually caught before things reach "release" trains, but I see it in bug trackers all the time.

2) Other layout engine managers I've talked with who want to do a layout system rewrite are terrified of anything that doesn't look like a large-scale refactoring because of this issue. I've had the "how are you testing and measuring progress / how do you know you're done with layout?" conversation with most of our engine peers at this point :-)

Re:Servo Nightly Builds Available by Manishearth on Friday July 01, 2016 @05:59AM (#12014867)
Well, we still do lots of reverse-engineering, because the specs are often wrong :)
Re:Servo Nightly Builds Available by gsnedders on Friday July 01, 2016 @06:21AM (#12014940)
A lot is relative. :)

Honestly, from what I see, the amount you do is relatively minuscule compared with what Opera was doing ten years ago, while Opera had far more marketshare which is normally the best way to avoid web compat issues in the first place.

Re:Servo Nightly Builds Available by nickpsecurity on Friday July 01, 2016 @10:50AM (#12016457)
It's a rule of thumb in most software I found studying high-assurance. Inevitably, some academic or commercial team wants to robustly implement some standard (esp a protocol). They notice it's specified in a combo of English and implementation code. They start doing formal specifications of English spec. Every time, IIRC, they find various inconsistencies and such that already did or could lead to real-world problems. Showed me the value of formal specs but also important lesson: using informal specs will almost always require reverse engineering and extensive testing due to, at very least, English ambiguity or implementation deviations.
Re:Servo Nightly Builds Available by MichaelGG on Friday July 01, 2016 @03:29PM (#12019014)
Implementing SIP a couple of times, part of it is the other implementations just flat-out misread. Even when there's a formal syntax specified, they simply ignore it. For instance, SIP specifies a "lr" (loose route) parameter. The syntax is just that: lr. But many implementations get it wrong and require a value, like lr=1, or lr=true.

Even things as simple as line endings are implemented wrong (in HTTP too), and that can cause security consequences as proxies end up reading headers differently than the user agents. I've seen this live on the public Internet.

SIP and the IETF to some extent encourages this with Postel's Law, telling implementors they should guess what the intention of the message is. We need less of VB's On Error Resume Next and more panic-abort type functionality. Look at this insane document: https://tools.ietf.org/html/rfc4475 "SIP Torture Tests". The authors gleefully come up with ridiculous yet legal permutations of messages that are allowed under their arcane rules. The fact this exists should send the opposite message: simplify your damn protocol. And this is only at the parsing level!

I know people say that HTML could never have had strictness because it'd have been too hard, but I don't buy that. One common issue was getting nesting wrong, like <b><i></b></i> or leaving unclosed tags. By removing the (silly, honestly) name out of the ending tag and just using </>, a whole class of errors is removed. Add in a browser that just fails to render and explains exactly why, and people would quickly not publish pages that are broken.

Re:Servo Nightly Builds Available by gsnedders on Friday July 01, 2016 @03:36PM (#12019071)
> SIP and the IETF to some extent encourages this with Postel's Law, telling implementors they should guess what the intention of the message is. We need less of VB's On Error Resume Next and more panic-abort type functionality.

I don't think it's as clear-cut as that. I think the main thing is error handling must be defined: it doesn't matter whether it's panic-abort or whether it's defined how to deal with any stream of bytes. Don't let implementers choose what they should do.

Re:Servo Nightly Builds Available by MichaelGG on Friday July 01, 2016 @07:10PM (#12020352)
Well the error handling is defined. Return a 400 Bad Request. Except implementors are encouraged to try to avoid that if they can make guesses.
Re:Servo Nightly Builds Available by gsnedders on Friday July 01, 2016 @09:30PM (#12020970)
It isn't everywhere.

RFC 7230, for example, has error handling sometimes as SHOULD return 400 Bad Request and sometimes as MUST return 400 Bad Request. But that only applies to servers and proxies parsing requests, which are really the relatively well-implemented part of HTTP. (The vast majority come from browsers, which are always syntactically valid; responses can come from arbitrary CGI scripts and you get all kinds of syntactic nonsense there.) Sadly, there's comparatively few normative requirements when it comes to parsing responses, even ones which are needed for backwards compatibility (at what point do you conclude you're just talking to an HTTP/0.9 server?).

Re:Servo Nightly Builds Available by Manishearth on Friday July 01, 2016 @12:59PM (#12017801)
So most of the spec bugs I've come across don't have to do with English vagaries; they have to do with the reverse-engineering not being perfect. Converting the nebulous concept of expected behavior into procedural text is hard, and there are bugs.

So yeah, "implementation deviations" is usually the issue, but not really English ambiguity, at least with web specs. I put my pedant hat on when looking at proposed changes to web specs, and I believe the rest of the community does too :)

Re:Servo Nightly Builds Available by gsnedders on Friday July 01, 2016 @03:33PM (#12019040)
There have certainly been spec bugs that could be found from a formal specification, some of which are security related (most of the TLS bugs found by miTLS have been protocol bugs simply found as a result of encoding much of the semantics formally).

That said, while I certainly have a fondness for formal specifications, I also always end up feeling like they have a lot of downsides. They're often very verbose and hard to skim, and they're also another language that anyone implementing the specification has to learn prior to implementing it (or they can just wing it, but then your formal specification is worth nothing!).

Re:Servo Nightly Builds Available by nickpsecurity on Friday July 01, 2016 @05:29PM (#12019810)
Re specs downsides

The wisdom from old days was to use relatively, simple, formal languages that were easy to train people on and use. B, Z, and TLA had good results plus lasted over time. The other thing was to do English spec, formal spec, and code side-by-side with each team looking for inconsistencies. The developers usually picked up the basics of the technique rather quickly but at least one specialist was always necessary to make first use go smoothly.

So, that's still my recommendation. A subset of those, especially focusing on states and contracts, done in same fashion against English specs and clear code should take lower work than going all Isabelle, NuPRL, or PVS on a problem. Second application is also always easier. Reuse where possible helps as with code.

So, some difficulties and training but probably easier than some frameworks, etc IT people learn today.

Re:Servo Nightly Builds Available by gsnedders on Saturday July 02, 2016 @05:43AM (#12021975)
(As a disclaimer, for some context, I have only relatively passing experience with Z, and the majority of formal modelling of specs I've done is using CSP.)

From when I've considered trying to formalise parts of the platform before, some of the things that often get hit very quickly are the fact that most of the web platform is defined in terms of sequences of Unicode code points, and ultimately any formalisation is going to have to deal with that (be it some complex parser—like HTML—or some simple validation of it matching a regular grammar). It has always seemed to me that to get much advantage you need to be able to programmatically assert statements about it (including, most obviously, pre-conditions are satisfied at all call-sites), but once you've gone from 2^256 to 2^1114112 characters you end up with such massive state explosion that even many symbolic tools start to struggle quickly.

Now, maybe I've tried using the wrong tools, or maybe I'm structuring things wrong, but it'd be nice to have some way to do this.

Re:Servo Nightly Builds Available by nickpsecurity on Saturday July 02, 2016 @10:56AM (#12022772)
Hmm. That's unusual situation. I've seen descriptions that sound similar in B data validation. For now, I'll just make a note of this comment to bring up when I next bump into formal methodists that might have answers.
Re:Servo Nightly Builds Available by gsnedders on Saturday July 02, 2016 @05:37PM (#12023875)
Thanks! If you get an answer, please drop me an email ([email protected]), because I'd love to hear it.
Re:Servo Nightly Builds Available by nickpsecurity on Friday July 01, 2016 @02:08PM (#12018362)
Interesting. Thanks for the perspective.
Re:Servo Nightly Builds Available by RubyPinch on Friday July 01, 2016 @10:04AM (#12016071)
There are some smaller groups that are working on their own engines (slowly but surely)

https://gngr.info/ comes to mind

Re:Servo Nightly Builds Available by nickpsecurity on Friday July 01, 2016 @05:37PM (#12019873)
If interested in Java browsers, then you might like to check out one that had quite a bit of development.

https://sourceforge.net/projects/xamj/

The main site is down due to dain issues. Googling will also give you a blog and Github. Their Cobra engine was Java IIRC. All that code might make a nice head start even if this project is dead or goes that way.

Re:Servo Nightly Builds Available by userbinator on Friday July 01, 2016 @12:42PM (#12017643)
There's also Dillo and Netsurf.
Re:Servo Nightly Builds Available by yellowapple on Sunday July 03, 2016 @02:17AM (#12025005)
Not to mention the various terminal browsers, like lynx and links.
Re:Servo Nightly Builds Available by coldtea on Friday July 01, 2016 @11:33AM (#12016930)
>I would say that since the late 90ies, there hasn't been a single new engine being made. Everything we have now is a development based on KHTML, Gecko, Presto or Trident.

Compared to KHTML, Webkit and Blink were pretty much 5 or more whole new projects on top of it. And even the latest IE is hugely rewritten compared to Trident.

Re:Servo Nightly Builds Available by pcwalton on Friday July 01, 2016 @11:13PM (#12021212)
There's a big difference between "slowly modified from something that works while keeping tests and web compat passing" and "written from scratch".
Re:Servo Nightly Builds Available by _pmf_ on Friday July 01, 2016 @05:25AM (#12014768)
Edge is a new engine, too.
Re:Servo Nightly Builds Available by iso-8859-1 on Friday July 01, 2016 @05:31AM (#12014786)
Re:Servo Nightly Builds Available by rat87 on Friday July 01, 2016 @05:32AM (#12014793)
doesn't seem like it

https://en.wikipedia.org/wiki/Microsoft_Edge#EdgeHTML

> EdgeHTML is a proprietary layout engine developed for Edge. It is a fork of Trident that has removed all legacy code of older versions of Internet Explorer and rewritten the majority of its source code with web standards and interoperability with other modern browsers in mind.

Re:Servo Nightly Builds Available by ybx on Friday July 01, 2016 @08:38AM (#12015491)
Sounds like it's based on Trident to me
Re:Servo Nightly Builds Available by coldtea on Friday July 01, 2016 @11:34AM (#12016936)
Sounds like it's so rewritten compared to Trident, it's not even alike anymore.
Re:Servo Nightly Builds Available by viscanti on Friday July 01, 2016 @12:04PM (#12017260)
The end result might look fairly different, but I'm not sure that 'forked-and-refactored' is considered 'new' by very many people.
Re:Servo Nightly Builds Available by nickpsecurity on Friday July 01, 2016 @12:08PM (#12017294)
"rewritten the majority of its source code with web standards and interoperability with other modern browsers in mind"

I think coldtea is focusing on highlighted part. A rewrite like that might change so many behavioral aspects, esp user- or developer-visible, that it can be considered a new engine rather than mere clean-up of behaviorally-identical old one. I certainly reuses some code and techniques but differences might be major.

Re:Servo Nightly Builds Available by kbenson on Friday July 01, 2016 @01:26PM (#12018015)
At the same time, depending on how the rewrite was carried out, it may be nothing like a completely new project. Rewriting function by function is different than rewriting component by component, which is also different than rewriting from scratch.

Rewriting function by function will give you largely the same program using largely the same algorithms, but possibly with more or less bugs, or cleaned up code.

Rewriting component by component will allow new ways those components are implemented, but the interface between components will likely be the same.

Rewriting from scratch could yield anything (servo could be classified as a way to rewrite gecko from scratch, I believe).

All of these could be said to have had the majority of their source code rewritten, but I would only really consider the last one to be "new" without knowing further details. Of course, the devil is in the details...

Re:Servo Nightly Builds Available by nickpsecurity on Friday July 01, 2016 @02:08PM (#12018356)
It's why I'm focusing on behavioral aspect: what does the software do? A browser fork that behaves very differently from prior one to point you have to change existing code to get same effect is essentially a new app at behavioral level. However, it might have same name and lots of same code inside. At that level, it's old code in an old app. We largely define our components by their interfaces and behavior. So that's what I'm focusing on.
Re:Servo Nightly Builds Available by kbenson on Friday July 01, 2016 @02:47PM (#12018688)
Sure, but if it behaves significantly differently, then I think that precludes it from being a function by function rewrite. At the same time, a component by component rewrite can be a complete rewrite (but it depends a bit more on the details, I think). If the mainly user visible portions are rewritten, but much of the utility code and other components are not, it may appear behaviorally different, but but be largely the same code base. It's much harder to know, looking in from the outside.

For a simplistic example, say someone provides a "rewrite" of grep. Maybe the options are all different looking and sounding, so it may appear to be significantly different. But if the core of the matching algorithm and it's capabilities are largely unchanged, is it a that much of a rewrite? To the outside user it may superficially appear so, but to someone comparing the source from before and after, there may be an entirely different opinion, and even that may change if you come from a context of focusing on a particular aspect.

As applied to Trident/Edge, we may have a case where the person speaking was involved in a project to rewrite one major component of the browser, and so is speaking in that context. Maybe that component is responsible for about 60% of the code and functionality, but it still relies on a quite a bit of largely unchanged additional libraries. It's very subjective as to whether you think that qualifies as a rewrite of the project, and depends quite a bit on what was not rewritten, and what you think of that code.

Re:Servo Nightly Builds Available by nickpsecurity on Friday July 01, 2016 @05:19PM (#12019728)
Re rewrite differences

Let's look at the grep example agsin. If it did same thing, but new interface, then it would be same thing because behavior is otherwise the same. Now, lets say you changed what pattern-matching itself produced where you no longer got the same results from same text. Matched totally different patterns unless you changed the text you are inputting to get same results again. And the interface was different. Is that still grep?

If he's talking one component, maybe not anything new underneath. He said the engine (rendering) would be more standards compliant and compatible with other browsers. That's basically what Opera and Mozilla did many years ago while IE didnt. If you blocked out the name, nobody using the browser would think they were using same app when they saw rendering hit those aspects totally distorting the presentation. Or had to redo their websites to display correctly. I remember reading many complaints from web developers about making stuff work with several, different engines whose rendering argued.

So, if Edge engine is different enough to cause that, then it seems like a different engine at behavioral level given same input leads to different output that fails acceptance. Nobody would think it was same app unless the UI told them.

I mean, I still think sameness or newness is an open issue. Do we judge it be function and component as you added? Or interface and behavioral spec as I was looking at? One might also look at data formats. I think it will best to technically just compare in various ways to be accurate. Feature-driven development field probably has some insight into this. However, users will look at them as different, at least in a version since, if they can't do what they were used for or break compatibility. The two Python versions are possibly a good example.

Re:Servo Nightly Builds Available by kbenson on Friday July 01, 2016 @07:15PM (#12020375)
> Let's look at the grep example agsin. If it did same thing, but new interface, then it would be same thing because behavior is otherwise the same.

That sort of depends on how you define "behavior". More specifically, if it matches something equivalent but different to regular expressions (even if it's just a character transliteration to regular expression syntax), then does the behavior change? Is that still grep? Maybe, if grep decides the the "new grep2 have an easier to learn matching language". It's not like we haven't seen that before.

> I mean, I still think sameness or newness is an open issue.

Sure! The vast majority of cases it probably is an open issue, because different people have different criteria. I was just pointing out that the wording from MS doesn't necessarily imply a rewrite as many people might define it, because we have very little context to go on in the statement, and different people have different ideas about what is the browser. The rendering engine? The UI? The JS VM? All of them combined?

I can totally see someone responsible for the rendering engine saying "we rewrote most of the source code" and the JS VM guys scratching their heads wondering what the hell they are talking about, because they definitely didn't rewrite most their code, and they think their component is a significant portion of the browser...

Re:Servo Nightly Builds Available by pilif on Friday July 01, 2016 @05:06PM (#12019645)
The last few IE exploits also affected Edge. I would say that there is still quite a few IE pieces left in Edge
Re:Servo Nightly Builds Available by anonbanker on Friday July 01, 2016 @02:20PM (#12018459)
This comment is a perfect example why arguing over proprietary software is mostly conjecture: nobody can look at the code and see what's different.
Re:Servo Nightly Builds Available by nickpsecurity on Friday July 01, 2016 @05:40PM (#12019885)
I'll mostly agree with that. Harder to do looking at assembler. However, at behavioral level, you can just black box it to see if it passes acceptance tests for desired outputs. If it does, it might be same for your needs. If it doesnt, it's clearly a problem.

Your point applies totally if we're talking the How instead of the What.