If I'm not mistaken, you can use the installation guidelines on swift.org, and just skip the Xcode related parts. I believe you just need to run the downloadable packages and install them, and they shouldn't require Xcode or anything. Then just add Swift's bin folder to your shell path.

https://swift.org/download/#apple-platforms