News for hackers. Stuff that matters.
Re:Rust required to build Gecko by userbinator on Friday December 16, 2016 @04:08AM (#13192027)
Gecko is already dependent on a lot of things so adding Rust might not be that big of a change, but IMHO a long and large list of dependencies and complex build processes are what can really put off those wanting to contribute to browsers (among other software) by fixing bugs or whatever else. I have personally experienced a few times where it was easier and faster to just patch the binary than figure out how to compile everything from source and go through the whole configure/install/etc process again.

Especially with browsers, which not everyone agrees on how they should be and desires to customise, only to find that the option to do so has been removed or a source change must be performed, is subsequently delighted to know that it's open-source so they should be able to do it easily, but then get overwhelmed and give up after they realise the effort needed just to build an unmodified version of the software themselves. They then fall back to merely complaining on the Internet, and reluctantly accepting their "fate"... somehow, I feel like some of the visions of open-source didn't quite turn out as well as hoped.

Re:Rust required to build Gecko by gcp on Friday December 16, 2016 @05:05AM (#13192227)
This is ironically something that rust (and cargo) solve, isn't it?

That said, what you describe isn't really an issue with Firefox. Unless you're running a very odd distro, apt-get build-dep or mach bootstrap will set you up automatically, and on Windows there's an installer with the dependencies (outside MSVC). Of coure, the people who like to customize to no end might be those running an odd distro, but something something digging yourself out of your own hole.

Re:Rust required to build Gecko by toni on Friday December 16, 2016 @07:42AM (#13192841)
You know, if the way you wrote your comment wasn't that much condensing, the people who like to customize to no end might learn from it.
Re:Rust required to build Gecko by gcp on Friday December 16, 2016 @07:49AM (#13192868)
Huh? My point was that if you changed your environment you probably understand what changes broke the automation and are able to fix it yourself.
Re:Rust required to build Gecko by toni on Friday December 16, 2016 @08:00AM (#13192922)
You gave some pretty great tips for using right tools, but (at least for me)it sounded like a grumpy nerd.

You could write your comment without accusing tweakers of digging themselves out of their own hole and all that unnecessary tone. In that case it would have more power to teach people something new.

Re:Rust required to build Gecko by reitanqild on Friday December 16, 2016 @08:23AM (#13193025)
He was most likely aiming at someone like me and I'm totally OK with it.

In fact I'm more scared by people trying to silence him/her.

Re:Rust required to build Gecko by toni on Friday December 16, 2016 @08:30AM (#13193073)
The grandparent was complaining about the complexity of building Firefox. Something that I truly sympathize with.

I certainly didn't want to silence the parent, but tried to point out that having a much more helpful tone will reach his message much farther.

Re:Rust required to build Gecko by koverstreet on Friday December 16, 2016 @11:02AM (#13194143)
Did you wake up on the wrong side of the bed or something?

Maybe reread this comment thread later, and compare the tone of the comments.

Re:Rust required to build Gecko by disordinary on Friday December 16, 2016 @03:32PM (#13196503)
gcp seemed perfectly reasonable and offered helpful advice, I detected no trace of contempt in there at all.

But that's the problem with context and text. Emoji FTW.

Re:Rust required to build Gecko by skj on Friday December 16, 2016 @08:35AM (#13193097)
It's almost always a mistake to assume a tone for a comment on the internet.
Re:Rust required to build Gecko by hxegon on Friday December 16, 2016 @03:02PM (#13196268)
Something something only 40% of intent comes across in text (couldn't find source, if someone can, please link)
Re:Rust required to build Gecko by Manishearth on Friday December 16, 2016 @04:25AM (#13192081)
The user-facing build process for Gecko doesn't change as a result of this (or as a result of most build changes). It's pretty straightforward: mach bootstrap to fetch dependencies, mach build to build. Takes a while but that's not something that complicates the contributor experience; the build time is just annoying.
Re:Rust required to build Gecko by weaksauce on Friday December 16, 2016 @01:23PM (#13195417)
Unless you are changing c/c++ code you can use artifact builds which download most of the compiled bits and integrate your front end changes. For me it changes a 1+hr build down to a <1min build.

It's basically one line to add to a config file to enable it.

Re:Rust required to build Gecko by Manishearth on Friday December 16, 2016 @01:29PM (#13195478)
Yeah, I'm really happy that this happened! I've been wanting it for years, ever since I started poking at Firefox frontend (not C++) code.

I actually had a proposal for a tool that would let you hack directly on your Firefox install without needing to download the full source (which is hard to do on flaky connections and annoying on slow ones). This is already possible, Firefox ships with a zip file (omni.ja) containing its UI code. The UI code is precompiled and cached, however you can edit the code and delete its cached form and stuff just works.

It's a hack, but it makes it incredibly easy to just get started hacking on Firefox code. I could imagine there being an addon that helped smooth this process.

Re:Rust required to build Gecko by hashhar on Friday December 16, 2016 @02:51PM (#13196181)
Could you point me to some resources? This really piqued my interest.
Re:Rust required to build Gecko by Manishearth on Friday December 16, 2016 @02:57PM (#13196219)
For omni.ja? http://inpursuitoflaziness.blogspot.com/2014/01/editing-file... is something I wrote a few years ago.

For more info about contributing to Firefox, see https://developer.mozilla.org/en-US/docs/Mozilla/Developer_g...

Re:Rust required to build Gecko by hashhar on Friday December 16, 2016 @03:11PM (#13196335)
I am about to start contributing. I currently keep reading the changesets to get an idea of how things work and hang out on irc and bugzilla. I am just waiting for my winter vacations to start working of Mozilla stuff. Thanks for the links.
Re:Rust required to build Gecko by ben0x539 on Friday December 16, 2016 @08:09AM (#13192970)
Rust seems like somewhat of a strategical direction for Firefox, not just any old frivolous dependency like mng support, so I think not prioritizing the brevity of the dependency list is reasonable here.

Have people traditionally been running their own modified browsers? Seems a bit scary given how you'd have to be on the ball to merge security fixes into your branch regularly.

Re:Rust required to build Gecko by gcp on Friday December 16, 2016 @08:14AM (#13192988)
There's a few forks that maintain the old UI that are popular, such as Pale Moon. They get security updates because the old ESR branches they're based on still get security patches from Mozilla.

Of course, that feed of patches will stop at some point. Don't ask me what the contingency plan for those forks is...

Re:Rust required to build Gecko by rebelwebmaster on Friday December 16, 2016 @11:42AM (#13194490)
IIRC, Pale Moon is based off a fork of ESR24, so they're already on their own for any security fix backports at this point.
Re:Rust required to build Gecko by Dylan16807 on Friday December 16, 2016 @03:58PM (#13196742)
I understand the impulse, but you can put the tabs back with a blob of CSS and leave the menus turned on and it's about the same. No need to use a fork for that.
Re:Rust required to build Gecko by cryptarch on Friday December 16, 2016 @07:50AM (#13192873)
How far do you think something like Docker/Vagrant build images would go to solve that problem?

I imagine an image with the full toolchain installed, geared towards using it as a kind of "firefox-compiler" CLI (so coupled with a few shell scripts), that only requires you to know how to install Docker|Vagrant. This could easily be bundled with the source code.

Re:Rust required to build Gecko by lmm on Friday December 16, 2016 @11:21AM (#13194271)
I think that would be more overhead, at least to the kind of people who have the native-code experience to be able to contribute to Firefox. (As more of the browser becomes Rust-based it will hopefully be more accessible for contribution, but once you're in the Rust world Cargo solves your problem a lot better than Docker does). And what do you do about the people left behind by Docker's extremely limited platform support?
Re:Rust required to build Gecko by vidarh on Friday December 16, 2016 @12:47PM (#13195133)
A Dockerfile is basically a set of instructions to run to produce a working image. Nothing prevents you from running those instructions outside of Docker. And nothing prevents you from structuring your Dockerfiles like:

    FROM suitable-base-image
    ADD ./install.sh /tmp
    RUN chmod a+rx /tmp/install.sh && /tmp/install.sh
and write a script to install the dependencies that people can opt to run outside of Docker if they prefer.

The point is not to take away peoples ability to pull together all of the dependencies on their own, but to make replicating the build environment trivial.

Personally I don't want install scripts etc. polluting my server or laptop setups - I always spin up containers to run builds in these days, because it means I at the end have a repeatable description of how to set it up. But once someone has done that job once, it's silly for everyone else to have to repeat it.

Re:Rust required to build Gecko by lmm on Monday December 19, 2016 @06:41AM (#13210818)
Install scripts were always the most opaque and difficult to use way to get a piece of software. The problem with Docker is that it encourages an opaque style of dependencies where you stop wondering or tracking which things are required and which are accidental. You stop depending on "foolib version 2.3 or later" and start depending on "foolib version 2.4.1p2". You stop being able to export the dependency graph and inspect it, because all you can do with a dockerfile is run it.

Docker maybe makes sense as a "compilation target" - generating a dockerfile of your dependencies to make it easy for people to use makes sense. But it's not rich or structured enough to be the canonical record of what dependencies your project needs.

Re:Rust required to build Gecko by vidarh on Monday December 19, 2016 @11:06AM (#13212266)
> But it's not rich or structured enough to be the canonical record of what dependencies your project needs.

I'm not arguing for it to be that, and I largly agree with you.

As my example shows, the Dockerfile doesn't need to contain any detail of the dependencies - you can "outsource" that entirely to whatever mechanism you wish, as long as you can easily add whatever you need to apply those dependencies to a suitable base image to the Dockerfile.

But most of the time I don't care about precision - I just want a means of getting a working build environment quickly. If it pulls in 100MB of unnecessary libraries, I don't care, if it gets me an uncomplicated way of building the project without having to assemble the dependencies myself.

For a lot of projects I come across, my experience is that because the list of build dependencies are often never automatically tested, it's a crapshoot whether or not their build instructions and dependency list will yield a working build environment on any given system.

"Even" a plain Dockerfile that is regularly used to rebuild the build environment is far superior to a text file of dependencies that is rarely to never verified.

If you want to go one better and storing more precise instructions and then use them to generate a build script or Dockerfile, then awesome.

The main thing is that absent automated testing of the build environment too, the list of dependencies isn't worth the bytes they're stored in. And a simple way of regularly testing the build environment is to rebuild it automatically.

My experience is that the dependency lists for projects that automatically rebuild the build environment - whether as Docker containers or in a chroot or any number of other ways - tends to be far more reliable.

And if you first can automatically build the build environment, it makes sense to make it easy for everyone to use that automatically built build environment.

Re:Rust required to build Gecko by vidarh on Friday December 16, 2016 @11:10AM (#13194188)
My thought too. I write the build processes for all my newer software explicitly as a two stage process: A Dockerfile with all the dependencies, including pre-requisite Makefiles etc., and a simple Makefile that has targets to (re-)build the build container and run builds in it.

If people want to install all the build dependencies themselves, that's fine, but there's really no reason to make people go through that hassle. Automating and isolating it also makes it far more liely that the dependency list remains accurate.

Re:Rust required to build Gecko by hashhar on Friday December 16, 2016 @02:52PM (#13196189)
I actually published such an image. You can find it at https://hub.docker.com/r/hashhar/mozilla-build/

It only includes the build tools though. You need the source code separately. I think I have provided enough info in the README to get you going.

Re:Rust required to build Gecko by annnnd on Friday December 16, 2016 @07:54AM (#13192899)
My thoughts exactly. Whenever I tinker with some new software I try to do it in a Docker container (writing all relevant commands to Dockerfile) so I can remove the changes easily when I want to.
Re:Rust required to build Gecko by zimbatm on Friday December 16, 2016 @09:12AM (#13193308)
One part of the solution might come from using Nix;

The release engineering team has started using Nix in their build process. Once all dependencies are mapped in Nix it means you just need to install the tool itself and all the dependencies will get pulled automatically.

It doesn't solve porting to new arch and platforms (especially Windows) so it's still a good idea to keep the number of dependencies down. But on any amd64 Linux it should be fine.

Re:Rust required to build Gecko by disordinary on Friday December 16, 2016 @03:30PM (#13196481)
I kind of think the opposite. A browser is probably the most technically challenging and complicated project out there, if you can't deal with dependencies then chances are your code will not be of sufficient quality to make it into the project that still makes up the whole. However, if it's split into a lot of dependencies then it's much more accessible because you can contribute to a smaller and more easily understood project.

In this case it looks like you can contribute to one of the servo project packages and those changes will make it downstream into Firefox, that's great as Servo has a (relatively) small and easily understood code base!

Re:Rust required to build Gecko by yeukhon on Friday December 16, 2016 @09:06AM (#13193277)
Re:Rust required to build Gecko by rockdoe on Friday December 16, 2016 @09:45AM (#13193529)
1) It's a request for comment. 2) This isn't really nodejs as it's wrapping the existing JS engine. They're not adding V8.
Re:Rust required to build Gecko by fenollp on Friday December 16, 2016 @09:36AM (#13193462)
Time to implement this idea:

Use open source projects' CI file to find which dependencies / commands are needed to compile / run the project. Run all this inside whichever container the CI system uses (circle & travis use only a few).

The issue described will never be an issue again!

Re:Rust required to build Gecko by witty_username on Friday December 16, 2016 @10:31AM (#13193896)
Or, apt basically.
Re:Rust required to build Gecko by pawadu on Friday December 16, 2016 @04:18AM (#13192062)
We call this the OpenOffice syndrome
Re:Rust required to build Gecko by slacka on Friday December 16, 2016 @04:54AM (#13192194)
I guess you haven't built it a decade or so(if ever). It's literally this easy to build:

$ git clone git://anongit.freedesktop.org/libreoffice/core

$ apt-get build-dep libreoffice

$ ./autogen.sh && make

With any relatively standard Debian/Ubuntu distro, you can have your own custom build in about an hour or so. Progressive builds just a few min if you have ccache setup.

Re:Rust required to build Gecko by cesarb on Friday December 16, 2016 @05:57AM (#13192392)
Yeah, but that's LibreOffice. The other OpenOffice descendant (Apache OpenOffice) still uses the old build system, and is still very complicated to build as a result; just look how long it took for its developers to release the 4.1.3 version, which was a very minor patch release.
Re:Rust required to build Gecko by fermigier on Friday December 16, 2016 @06:07AM (#13192436)
Apache OpenOffice is on life support (at least, was in September, I wonder if the situation has evolved since):

https://lwn.net/Articles/699047/

Re:Rust required to build Gecko by banachtarski on Friday December 16, 2016 @05:57AM (#13192393)
An hour is pretty awful for a word processor. I can build a fully modern 3D engine in half that time.

edit: I noticed you typed make. Do you perhaps mean make -j 8 or something (depending on core count)

Re:Rust required to build Gecko by Manishearth on Friday December 16, 2016 @01:23PM (#13195419)
An hour makes sense to me.

Here's what I can think of off the bat:

- Handling fonts. A whole world of hurt. Not all of this can be offloaded to OS libraries.

- Handling layout. Another world of hurt. Especially given that many formats have their own quirks.

- Handling complicated things like math and svg.

- Understanding HTML (for formatted paste and save-as-html to work)

- Understanding PDF

- Canvas for the shapes and stuff

- Codecs and stuff for images and video

- General rendering/compositing code

- Accessibility code

There's a LOT that goes into building software like this.

I worked on Microsoft Office as an intern once. The codebase is huge. Build times would be forever without artifact builds. I currently work on browsers, which are in the same situation with a lot of not-immediately-obvious costs.

Re:Rust required to build Gecko by banachtarski on Friday December 16, 2016 @10:54PM (#13199094)
Graphics engines handle fonts. Even more so than a word processor. They have to AA correctly at all viewing angles, textured, upscale, downscale, animate, etc.

Layout is in every modern game engine as UI and is even more powerful. See scaleform for example.

Math... isn't in a game engine?

HTML/PDF and other formats? Please how about DXT1-N texture formats, crunched textures, scene descriptions, meshes, animation rigging, etc. HTML and PDF are small in comparison.

Yea I'm not going to continue. What you may not have realized as a MSFT Office intern is that just because it was slow and awful, it might not have necessarily needed to be.

Re:Rust required to build Gecko by Manishearth on Saturday December 17, 2016 @01:50AM (#13199638)
> What you may not have realized as a MSFT Office intern is that just because it was slow and awful, it might not have necessarily needed to be.

It's not just Office. It's also the two browsers I have worked on (a word processor in particular has a lot in common with a web browser, but it also needs to deal with authoring). There are many things that need to be done that aren't immediately obvious. Sure, a graphics engine has such costs too but it's hard to compare unless you're well acquainted with both sides.

Re:Rust required to build Gecko by sanxiyn on Saturday December 17, 2016 @07:08AM (#13200405)
Math here means typesetting mathematical formula, which is in word processor and browser, but not in game engine. It's also quite complex.
Re:Rust required to build Gecko by IshKebab on Friday December 16, 2016 @06:26AM (#13192509)
I think that a word processor like OpenOffice is significantly more complicated than most 3D engines.
Re:Rust required to build Gecko by dom0 on Friday December 16, 2016 @06:38AM (#13192566)
It might be more code, and it surely has a lot of tricky code and shit-to-maintain code (.doc import?!), but I very much doubt it's more complicated.

That being said complicatedness has little to do with compile time now, has it? :)

Re:Rust required to build Gecko by agentgt on Friday December 16, 2016 @08:20AM (#13193011)
> It might be more code, and it surely has a lot of tricky code and shit-to-maintain code (.doc import?!), but I very much doubt it's more complicated.

There are some articles of Joel Spolsky explaning why office document formats are so complicated (not my word) [1]. The spec for MS doc is like 500 or so pages. The spec is longer than my video game graphics book I had in college. I recommend some research before assuming.

> That being said complicatedness has little to do with compile time now, has it? :)

Actually there is a correlation [2]. Of course it depends on what you call or think complexity is.

Believe it or not engines may not actually be as complex because they typically are based on math formulas or combinations of simple algorithms. Math is often simple and doesn't require lots of branching (if conditions). The algorithms have to be fairly simple anyway otherwise performance would be bad.

Buisiness software or actual video games on the other hand are full of branching. NetHack may not have 3D graphics or any graphics but it is an extremely complicated game.

[1]: https://www.joelonsoftware.com/2008/02/19/why-are-the-micros...

[2]: https://en.wikipedia.org/wiki/Cyclomatic_complexity

Re:Rust required to build Gecko by yrro on Friday December 16, 2016 @08:08AM (#13192965)
It's not just a word processor, but an office suite. So you're also building the spreadsheet, drawing program, presentation program, maths editor, database program...
Re:Rust required to build Gecko by necessity on Friday December 16, 2016 @07:40AM (#13192829)
Uh, yes, it is a big change, Rust is huge. I for one am dropping this bloatware for a WebKit browser once the changes hit Gentoo.
Re:Rust required to build Gecko by pcwalton on Friday December 16, 2016 @11:32AM (#13194403)
Can you elaborate as to why Rust is bloat, and specifically how to achieve the goals of improved security and parallelism without using it?
Re:Rust required to build Gecko by shakna on Friday December 16, 2016 @07:59AM (#13192919)
V8 seems a bit more bloated to me than what comes bundled with Firefox.
Re:Rust required to build Gecko by Matthias247 on Friday December 16, 2016 @08:29AM (#13193065)
Maybe yes, maybe no. But Webkit doesn't use V8 anyway :)
Re:Rust required to build Gecko by shakna on Friday December 16, 2016 @01:17PM (#13195375)
True, I may have assumed they meant Blink, and something something Chromium, which breaking away from V8 isn't particularly easy, even if one is the CSS engine, and the other the JS engine. Sorry for leaping to conclusions.

But they may have meant Midori or similar which actually uses WebKit, through WebKitGTK+... Which means linking against GTK+, probably GTK+3 to get the main benefits like single-page not whole application crashing.

Which depending on your custom build, may or may not be an absolutely huge overhead.

It could be a tiny change to add to an OS. Or huge.

Re:Rust required to build Gecko by gcp on Friday December 16, 2016 @07:46AM (#13192856)
Do you realize this is a build-time, not run-time dependency?
Re:Rust required to build Gecko by loudmax on Friday December 16, 2016 @07:52AM (#13192891)
Do you realize that Gentoo packages are distributed by source code rather than pre-built binaries?
Re:Rust required to build Gecko by Manishearth on Friday December 16, 2016 @11:54AM (#13194623)
FWIW last I checked Chromium took way more time and RAM to build for the first time from source. This was many years ago.
Re:Rust required to build Gecko by SSLy on Friday December 16, 2016 @08:58AM (#13193228)
Except there are -bin packages there.
Re:Rust required to build Gecko by disordinary on Friday December 16, 2016 @03:50PM (#13196674)
I think once the transition to more rust components and parts of the servo project is complete the bloat will actually reduce significantly because Servo and all these libraries are built from the ground up without all the bandaids that have gone into decades of gecko development.
Re:Rust required to build Gecko by nilved on Friday December 16, 2016 @11:35AM (#13194431)
> I for one am dropping this bloatware for a WebKit browser

Oh come on