Apple actually uses a bunch of 3rd party developer tech on the server-side and on the web. They use Rails quite a bit I believe and are starting to use Angular.js too.
My guess would be that the webcrawler was started before Swift was allowed to escape from the Developer Tools enclave.
Surely you agree that languages designed with one purpose in mind (a la Go and networking/server development) can reasonably do better at that purpose than an explicitly general-purpose language?
Just to be clear, I prefer Swift to Go in general. However, between the two, I would personally prefer to use Go for the development of network utilities.
I don't think that's a given. Just because a language is more special-purpose, doesn't mean that a more general purpose language can do the same things as seamlessly as in the other language (maybe they would have to define an embedded DSL, but that might be simple). Maybe the special-purpose language decided to commit to certain things in its domain - like certain concurrency features - and make it "first class", since that was its intended domain. But then maybe that concurrency feature wasn't that useful and a more general purpose language - which had to keep its options open since it was general purpose - could express other alternative features more easily.
Even as an outspoken critic of go, I recognize that it has quite a strong networking toolset.
Using the right language for a particular domain can save you a lot of headache. Go is open source and a relatively proven choice, compared to Swift.
As for access to the language implementation, Swift is Apple's creation.
I'm not trying to tell you that you are wrong. I'm just saying that your reasoning is flawed, and your arguments don't really support any particular conclusion. "Swift is a general purpose language." - OK? "I fail to see what it lacks." - Noted.
Let's only compare with native, canonically AOT compiled, general purpose languages, though. All of a sudden there are a lot of questions. Why didn't they write it in C? Why didn't they write it in Delphi? Fortran? Forth? The answer is of course that beyond being a native, canonically AOT compiled, general purpose language, different languages have different constructs and models of particular patterns that make them more or less useful for different kinds of tasks and development models.
Apple seemingly recognized that Swift was less useful than Go for the particular task at hand, and while we can't really know for sure if this was a matter of performance, development speed, personal preference of the developers etc, we can probably assume that the choice was based on taking several such quality dimensions into consideration.