High up on this list is poor work/life balance, that the company is associated with (worse even than the typical awful work/life balance in aerospace at-large). Not wanting to deal with the absurdly unrealistic deadline or engineering expectations that the company is known for is a close second.
In the interest of full disclosure I did interview with SpaceX and was rejected due to a lack of experience. I don't think this colors my opinion too much, as most of the above complaints come from talented engineers who have been working in the industry much longer than I.
I guess I'd wonder whether SpaceX would ever intentionally target those people for "poaching" though given they're not a cultural fit to begin with? It would appear from the outside to be quite a self selecting situation?