They do use it internally and a lot of teams do complain about it. I hope there is some real work going on to improve some parts of it (how about shipping patches rather than an entire 11gb binary every version). I think a lot of devs also pin certain issues on Xcode that are actually issues with the Swift compiler (slow or missing autocomplete, unhelpful error messages or slow compilation times for example) ObjC is far far quicker to compile.