Re:Testing LLVM
by KenoFischer on Friday January 13, 2017 @04:12PM
(#13394745)
I always find committing to LLVM very nerve wrecking, because of the post-commit CI testing. LLVM has so many architectures that more often than not something I write will fail on one of them. And the only way for me to find out is to commit it, wait for the buildbot to fail (which can take a few hours, during which I really can't leave my computer lest I leave trunk broken on some buildbot, which is a big faux pas), revert it and then figure out what went wrong. I'm hoping that at some point this will be improved, such that I can run the whole buildbot army on my commit before putting it on trunk.
[ Parent
| on HN ]
Re:Testing LLVM
by shanemhansen on Friday January 13, 2017 @05:23PM
(#13395306)
That's odd. It would be nice if branches could be tested with the build bot.
I agree. When I submit a commit for Racket, the branch is tested in Travis and AppVeyor. That catches a lot of error.
Anyway, Racket has an additional internal CI called DrDr and some very subtle errors are only detected there, after the commit is merged.
I get the feeling that there's parts of the community that feel the same way. I'm hoping that the planned move to github will naturally cascade into pre-commit checks.
A big big problem is having enough hardware to have a CI throughput one or two order of magnitude what it is now. Unfortunately that's not an easy thing considering how "heavy" it is to build and test the full toolchain.
Do you know of solutions that exist for pre-commit checks for GitHub at the moment?
Re:Testing LLVM
by dikaiosune on Friday January 13, 2017 @05:26PM
(#13395326)
Rust uses https://github.com/rust-community/bors which maintains a linear queue of PRs which can only land after being rebased onto the commits before the PR and subsequently passing tests.
Re:Testing LLVM
by WalterBright on Saturday January 14, 2017 @12:03AM
(#13396990)
dlang uses several. One notable one is the "autotester" written by Brad Roberts. It's built to ensure that breaking the D compiler (as tested by the test suite) does not result in commits.
It's incredibly useful.
Swift also has a lot of pre-commit CI:
- here is the Jenkins bots page: https://ci.swift.org/view/Pull%20Request/ - example of PR test reporting: https://github.com/apple/swift/pull/6802
In general? Yes, Travis CI seems to be a very popular one. Many folks use Appveyor for Windows support AFAICT.
Most of the .NET repos are structured to use inner and outer loop testing with Jenkins. Most tests on most architectures are run in the inner loop, which are kicked off in parallel as soon as you make a pull request to one of the .NET repositories on Github.
Some repositories, like CoreCLR, have outer loop testing that runs on a separate schedule (nightly, I think), but those tests are far less likely to break and are more devoted to finding rare and difficult to compute edge cases.