I've always found it interesting that Chrome is shipped mostly as one big binary (or on Windows, two: chrome.dll and chrome_child.dll), whereas IE and Safari use several (one for the rendering engine, one for the JS engine, one or two for networking, etc.). My guess is that Chrome uses one big binary to minimize startup time, since the OS can just map that binary into memory and go. Am I correct about the rationale? Another way to ask the same thing is this: what would be the downsides of using a component build for releases and not just for development? If you used a component build for releases, you'd have greater parity between dev and production, and that's generally a good thing.
I also notice that there are a few modules that get their own DLLs, at least on Windows. For example, in Chrome 45, we have chrome_elf.dll, libegl.dll, libexif.dll, and libglesv2.dll; I'd think these libraries could be statically linked just like the few hundred others. Are there good reasons for these exceptions? Or are these just little details that tend to get overlooked given the rapid pace of development?
As for the remaining DLLs, it mostly comes down to licensing concerns. We don't include proprietary code in our main binaries, and in order to simplify IP concerns it's often best to split such code into other DLLs. For example, you asked about libegl.dll and libglesv2.dll, which I believe include a licensed copy of the SwiftShader library.
It's also worth pointing out that the performance impact of having a single, static binary in Windows is much less significant than it used to be. So, were we starting Chrome today I'm not so sure we'd make the same decision.
But since I've seen @bengoodger in this thread, maybe he can answer your questions.
I'm not sure why there are now a handful of separate dlls.
Also, we are compiling and testing on Mac, Windows, Linux, Android, iOS, ChromeOS, and historical versions of all of the above. That's something that a mostly linux-server-based company doesn't do a lot of. So we maintain our own datacenters with our own hardware, and don't get to take advantage of all of Google's scale.
And FWIW on building MongoDB with Evergreen (https://evergreen.mongodb.com) we've found that a stepback approach seems to give us the finest granularity with minimal cost in identifying the introduction of an error - we batch commits for execution, but when a specific test fails we start running just that test on each previous commit until we hit a passing execution. It obviously doesn't work perfectly in the face of flaky tests (see above question) but it seems to do pretty well.
The first is automatic retries of fast tests. If a test runs quickly and fails, it costs us little to try again just to make sure. Most of our unit tests are configured to run up to 3 times.
Another thing is keeping track of a database of individual test case passes/failures across all time. This will let us automatically mark tests as flaky if they fail often, and ignore their results programatically rather than requiring a human to manually mark the test as ignorable.
A third thing is, obviously, automatically filing bugs against the owners/authors of tests which have been marked flaky. This is controversial -- often a test is just fine until one of its underlying libraries has a race condition introduced, and the real person to fix it should be the author of that change, not the author of the test. But it is still a step in the right direction much of the time.
Many people subscribe to the philosophy that "a flaky test is worse than no test", because you think it is giving you information when in fact it is giving you none. I subscribe to a slightly different philosophy: "A test with a known flaky rate is hugely valuable". If you know how often a test flakes (statistically), then you can measure variances from that rate to detect changes. Of course, a flaky test with an unknown rate of flaky is still useless. Hence the second initiative above: measuring the rate of flake of everything.
Firefox has an experimental "chaos mode" that takes the opposite approach. It purposely randomizes behavior by adjusting thread priorities, changing hash table iteration order, and randomize timer durations. Unfortunately, many flaky tests fail in chaos mode, so it is not enabled by default.
http://robert.ocallahan.org/2014/03/introducing-chaos-mode.h...
I think one of the lessons we've learnt is that with a browser-type project it's very hard to make test runs fully deterministic, for both technical and human reasons.
The technical reasons are touched on in the original article: these are complex codebases with lots of moving parts and lots of environmental dependencies. Of course there are various tactics to try and combat this; for example there is a wiki page dedicated to innocuous-looking code that leads to intermittent tests [1].
The human reasons centre around the difficulty of getting people to care about spending time fixing a test that fails one time in 1,000 (which is still very noticeable when you are running it hundreds of times a day). Unless the issue is something that fits a known pattern it's hard work, difficult to tell if your fix even worked, and not likely to be considered a top priority due to the diffuse, hard to quantify, nature of the benefits.
I think the fact that both Google and Mozilla still have significant problems with intermittents despite talented engineering staff and it having been a known problem for years implies that some of the standard thinking about making tests fully deterministic simply doesn't apply; for this kind of work you have to embrace — or at least accept — the randomness, and look for ways to get the data you need despite the noise.
[1] https://developer.mozilla.org/en-US/docs/Mozilla/QA/Avoiding...
Could you run "all" the identify flaky tests by running all the tests 100 times on the same stable build (like the latest ESR)? Is it even possible to write a test that could pass 100 times in a row? :)
Unfortunately there are various confounding factors that mean many intermittent tests would look clean in such a run might nevertheless be problematic. For example if you only run tests that you think are intermittent problems that are triggered by state left from a previous test won't be found. This is one reason that we've been trying to run particularly problematic test types (e.g. firefox browser-chrome tests) in smaller groups restarting the browser with a clean profile between groups to clear the state. A group size of 1 would obviously be ideal here, but when you have thousands of tests and limited resources it's not practical.
The other problem is tests that have unexpected sensitivity to the environment. For example the other day DNS was being slow on the test infrastructure. This isn't a problem for most tests since they use something like /etc/hosts. But some tests were intentionally trying to use a non-resolving domain and those tests sudden started to randomly time out.
I debugged those bugs with custom gdb scripts. The test system will run the gdb scripts to start the app. The gdb script setup breakpoints, log stack trace and local variables automatically for a particular bugs.
The debugging environment is scripted and automated. Test failure mark/track the log file with all the gdb info.
I schedule the tests to run continuously overnight and in the morning I analyze the log file with gdb info.
After that I can add more debug command in gdb script or try fixed the bug if I have enough info. The cycle continue until the bugs were all fixed.
I personally think it was very good for try to analyze / fixing the "flaky" tests issue.
Even for the the tests that fails 1 in 1000 times, the bugs can still be debug and fixable.
It was actually more relaxing for me to debug this kind of problem. 99% of time, the computer do all the works.
You guys might want to try it if not done in Google already.
More importantly, you seem to be running all the tests, as often as you can. A commit changing a text label shouldn't require running all the tests, that just doesn't scale. Have you consider splitting Chromium into different projects, where commits for one project would result in the tests of that project, and only that project?
We actually don't run all the tests all the time. We have a good dependency system (called Gyp/GN, similar to Bazel) which knows which tests rely on which source files. When the infrastructure compiles a new change, it only runs the tests which depend on source files touched by that change.
For example:
a) No merges; and the only branches are release branches. This makes sense in almost any context. Whether your team is big or small, high-caliber or college sophomores, this works. The advantages are numerous. No one works on a feature branch in isolation, only to have tons of merge conflicts and surprises when they try to bring it back in. Bug fixes and features are never landed on a release branch and then not merged into master for the next release. Development doesn't slow to a crawl while you try to stabilize master for a release.
b) Keeping dependencies near tip-of-tree. Chromium has two kinds of dependencies: third-party libraries (e.g. yasm, lighttpd) and things we develop on our own but in separate repositories (e.g. pdfium, v8, boringssl). The former tend to have stable releases produced by their own maintainers; the latter is being developed in parallel and you want to be as up-to-date as possible. Most projects are similar: you have third-party dependencies (e.g. python packages) and your own libraries (like a common set of api interfaces). You want to keep the ones you control as close to tip-of-tree as possible so that you can move fast and integrate your own changes quickly.
The caveat to all of this "move quickly" stuff is "test Test TEST". If you are constantly rolling dependencies forward, stuff will break. Run all the tests before landing a dependency change. Revert dependency changes with impunity: since they are effectively merges of many commits made to the other repo, they are more likely to have breaking changes than any single commit to your main repo. So pretest, test, and revert.
None of this requires amazing engineers. It just requires a dedication to quality and good testing hygiene.
Did I understand that correctly?
But yes we also provide tools (https://commondatastorage.googleapis.com/chrome-infra-docs/f...) to make things like rebasing (see git rebase-update) your local branches on top of master easier.
At my startup, we use feature branches, and we expect devs to continually merge master into their branch while it's under development (ideally daily), and to fix any resulting conflicts or test failures. We have them push these branches to the repo (which triggers our continuous integration server), but they're treated as private works in progress. Then, when they finish and pass code review, we're confident that they will merge seamlessly, since they've been merging master all along. (We also use git merge --squash for the final merge to avoid seeing all the intermediate commits on the master commit log.)
I'm trying to understand if your practices are meaningfully different. Do developers have private branches on their machines for weeks on end? Are they encouraged to regularly merge master into those private branches?
However, large portions of the team are opposed to Gerrit because a) it doesn't integrate as well with our other systems (such as trybots and cq) and b) the UI has some issues. We are working with people to try to fix those, but there's no particular timeline for switching, or even a guarantee that we will.
Is there a tracking bug for all the integration and UI issues? How can folks working on Chromium but without an @google.com help?
There are three things going on:
1) Rietveld exists
2) A new Polymer-based UI for Rietveld exists
3) Gerrit exists
The bug linked above is about evaluating the ability to make the Polymer-based UI (which is pretty and functional) work on top of Gerrit instead of on top of Rietveld. This is one of the possible paths forward for getting people to use Gerrit, and the one for which external contributors could be the most help.
Other paths forward exist: making Rietveld and the polymer Rietveld UI so good that no one wants to switch to Gerrit anymore; working directly on the Gerrit UI; or convincing devs that the Gerrit UI is good enough. But I think the bug and description above is the path on which external contributors could do the most good.
https://en.wikipedia.org/wiki/List_of_tools_for_static_code_...
I know in safety-critical the Astree and SPARK tools have been particularly good at verifying code. Something like Polyspace might be more appropriate for Chrome, though.
Instead, we've focused pretty heavily on more conventional anti-exploit mitigations applied to existing C++ code. Our sandboxing is the most obvious one, but we have a variety of other protections on top of typical stuff like ASLR and NX (e.g. heap partitioning, a safe integer handling toolkit, etc). We're also leading efficient control-flow integrity work in clang, and busily working it up to a stable release. And then we have fuzzing as part of our continuous integration, running across thousands of cores of instrumented builds.
Do keep the exploit mitigation tech, though. The NaCl paper was one of best reads I had that year. Most interesting one right now is Code Pointer Integrity [4]. Curious, is there a configuration that swaps out the sandbox for something with the same interface but no real sandbox? Point is if we can swap out the existing sandbox (esp CFI) for a different technology (eg CPI, Softbound + CETS).
[1] http://www.drdobbs.com/an-exception-or-a-bug/184401686
[2] http://www.codeproject.com/Articles/8293/Design-by-Contract-...
[3] https://webcache.googleusercontent.com/search?q=cache:WZAQPu...
[4] https://www.usenix.org/system/files/conference/osdi14/osdi14...
I mentioned PolySpace as an example of something more appropriate for Chrome because it supports C++ and is used in high integrity applications.
Even committers are heavily encouraged to use the commit queue, and everyone does. Something like 96% of commits land via the CQ. It's simply a bad idea not to. The main use-case for bypassing the queue is quickly reverting a change which somehow slipped through, and even those usually go through the queue anyway with a special flag that bypasses all of the tests.
I'd agree that having long-lived branches is often bad, but that's more a matter of discipline than anything else. I could certainly start working on a feature locally using Chromium's model, take my time, and get far enough behind the tip of the tree that by the time I want to submit for code review and testing, my stuff doesn't apply to the new tip.
Branches vs. no branches doesn't seem to be a factor here.
Yes, it is totally possible for two developers to be working in the same place and try to land their changes at the same time, resulting in a conflict. This happens in one of three ways: a) dev B's change fails tests, because the try bots which try to test dev B's change have already sync'd dev As change (this is the most common) b) dev B's change passes all the tests, but fails to commit because dev A's change landed seconds earlier. In this case the system sends dev B an email asking them to rebase their change and try again. Simple. c) dev B's change passes all the tests, then lands and causes tip-of-tree tests to fail because dev A's change landed seconds earlier and they disagree on the name of an API method or something. This happens rarely, and is easily resolved by reverting one of the two and asking the author to try again.
Anyway, the point is that the dev branches are incredibly short lived: they exist only on the developer's machine and end up as a single, coherent, self-contained commit on master. If you get far enough behind that your stuff doesn't apply on master, it is your responsibility to clean it up, and no one else even knows that you had a conflict.
Do developers ever want to collaborate on an in-progress feature before it's ready for master? As far as I can see the only real difference between your workflow and a github-flow-like workflow is that they can't do that because their feature branches are local.
There is no such thing as "not ready for master" in the chromium workflow, only "not ready to be enabled by default".
You could automatically bisect with just the failing test to find which commit to revert. At least if it's not a flapping test.
As he says, this is slightly out of date: we forked WebKit to make Blink in 2013, and actually merged Blink into Chromium proper just this week! It is no longer managed by gclient/DEPS, but instead developed alongside Chromium in the same repository. But many other things (pdfium, v8, webrtc, and tons of smaller libraries) are still pulled in via DEPS in exactly the way described.
I'd just like to say this about our development, testing, and release process: it really really works. We keep moving fast, everyone knows what's going on throughout the codebase because everyone is working on master, and there are never any big ugly merge conflicts to resolve when new features land.
There are, of course, problems with the system. We don't have enough resources to run every test for every commit, so they end up batched and we have to rely on humans and heuristics to determine which CL in the batch to revert. The set of configurations on the try-bots doesn't exactly match up with the set of configurations that are run against tip-of-tree. And flaky tests are, of course, a perennial problem for everyone. But we're working on making all of these better, and huge progress has been made in the last couple years.
Ask me anything if you have any questions about our build and test infrastructure!